mlsanityData to 7 Oct 2026

EAT: Self-Supervised Pre-Training with Efficient Audio Transformer

arXiv 2401.03497 ↗First posted 7 Jan 2024eess.AS cs.AI cs.CLcited by 52

Not on the map. Why?

Abstract (arXiv)

Audio self-supervised learning (SSL) pre-training, which aims to learn good representations from unlabeled audio, has made remarkable progress. However, the extensive computational demands during pre-training pose a significant barrier to the potential application and optimization of audio SSL models. In this paper, inspired by the success of data2vec 2.0 in image modality and Audio-MAE in audio modality, we introduce Efficient Audio Transformer (EAT) to further improve the effectiveness and efficiency in audio SSL. The proposed EAT adopts the bootstrap self-supervised training paradigm to the audio domain. A novel Utterance-Frame Objective (UFO) is designed to enhance the modeling capability of acoustic events. Furthermore, we reveal that the masking strategy is critical in audio SSL pre-training, and superior audio representations can be obtained with large inverse block masks. Experiment results demonstrate that EAT achieves state-of-the-art (SOTA) performance on a range of audio-related tasks, including AudioSet (AS-2M, AS-20K), ESC-50, and SPC-2, along with a significant pre-training speedup up to ~15x compared to existing audio SSL models.

Key facts. Cited by 52 papers in the map (data through 7 Oct 2026). Builds on 45 of its 51 references that are in the map. First posted on arXiv 7 Jan 2024.

Builds on

45 of 51 references.

References · 45

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  8. 2016Layer Normalizationcited by 5,579
  9. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  10. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  11. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  12. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
33 more
  1. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  2. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  4. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  5. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  6. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  7. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  8. 2016Deep Networks with Stochastic Depthcited by 973
  9. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 741
  10. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  11. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  12. 2021AST: Audio Spectrogram Transformercited by 634
  13. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 563
  14. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 378
  15. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  16. 2021AudioCLIP: Extending CLIP to Image, Text and Audiocited by 335
  17. 2022Masked Autoencoders that Listencited by 271
  18. 2021Efficient Training of Audio Transformers with Patchoutcited by 222
  19. 2021Wav2CLIP: Learning Robust Audio Representations From CLIPcited by 210
  20. 2022HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detectioncited by 174
  21. 2021SSAST: Self-Supervised Audio Spectrogram Transformercited by 166
  22. 2022Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Languagecited by 108
  23. 2021BYOL for Audio: Self-Supervised Learning for General-Purpose Audio Representationcited by 106
  24. 2022MAE-AST: Masked Autoencoding Audio Spectrogram Transformercited by 82
  25. 2021PSLA: Improving Audio Tagging with Pretraining, Sampling, Labeling, and Aggregationcited by 81
  26. 2022Masked Spectrogram Modeling using Masked Autoencoders for Learning General-purpose Audio Representationcited by 52
  27. 2022Masked Spectrogram Prediction For Self-Supervised Audio Pre-Trainingcited by 50
  28. 2023CED: Consistent ensemble distillation for audio taggingcited by 29
  29. 2023Self-supervised Audio Teacher-Student Transformer for Both Clip-level and Frame-level Taskscited by 25
  30. 2022Masked Modeling Duo: Learning Representations by Encouraging Both Networks to Model the Inputcited by 23
  31. 2021Conformer-Based Self-Supervised Learning for Non-Speech Audio Taskscited by 22
  32. 2022ATST: Audio Representation Learning with Teacher-Student Transformercited by 13
  33. 2022MT4SSL: Boosting Self-Supervised Speech Representation Learning by Integrating Multiple Targetscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 26; 2026: 19.

52 papers.

Sort:

By area:

  1. 2026MOVA: Towards Scalable and Synchronized Video-Audio Generationcited by 39
  2. 2024BirdSet: A Large-Scale Dataset for Audio Classification in Avian Bioacousticscited by 36
  3. 2024BAT: Learning to Reason about Spatial Sounds with Large Language Modelscited by 34
  4. 2024Foundation Models for Music: A Surveycited by 34
  5. 2024MuPT: A Generative Symbolic Music Pretrained Transformercited by 32
  6. 2025Can Masked Autoencoders Also Listen to Birds?cited by 26
  7. 2025SSLAM: Enhancing Self-Supervised Models with Audio Mixtures for Polyphonic Soundscapescited by 16
  8. 2025Token Pruning in Audio Transformers: Optimizing Performance and Decoding Patch Importancecited by 12
  9. 2025Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learningcited by 12
  10. 2025Foundation Models for Bioacoustics -- a Comparative Reviewcited by 11
  11. 2024DRCap: Decoding CLAP Latents with Retrieval-Augmented Generation for Zero-shot Audio Captioningcited by 10
  12. 2025TARO: Timestep-Adaptive Representation Alignment with Onset-Aware Conditioning for Synchronized Video-to-Audio Synthesiscited by 10
38 more
  1. 2025USAD: Universal Speech and Audio Representation via Distillationcited by 7
  2. 2025Unmute the Patch Tokens: Rethinking Probing in Multi-Label Audio Classificationcited by 7
  3. 2026SLAM-LLM: A Modular, Open-Source Multimodal Large Language Model Framework and Best Practice for Speech, Language, Audio and Music Processingcited by 7
  4. 2025AnyCap Project: A Unified Framework, Dataset, and Benchmark for Controllable Omni-modal Captioningcited by 5
  5. 2025AVEX: What Matters for Animal Vocalization Encodingcited by 5
  6. 2025PAL: Probing Audio Encoders via LLMs -- Audio Information Transfer into LLMscited by 3
  7. 2025ViDove: A Translation Agent System with Multimodal Context and Memory-Augmented Reasoningcited by 3
  8. 2025FISHER: A Foundation Model for Multi-Modal Industrial Signal Comprehensive Representationcited by 2
  9. 2025Cleaning the Pool: Progressive Filtering of Unlabeled Pools in Deep Active Learningcited by 2
  10. 2024Towards reliable respiratory disease diagnosis based on cough sounds and vision transformerscited by 1
  11. 2024Bridging Speech and Text: Enhancing ASR with Pinyin-to-Character Pre-training in LLMscited by 1
  12. 2025AC/DC: LLM-based Audio Comprehension via Dialogue Continuationcited by 1
  13. 2025ASDA: Audio Spectrogram Differential Attention Mechanism for Self-Supervised Representation Learningcited by 1
  14. 2025An Enhanced Audio Feature Tailored for Anomalous Sound Detection Based on Pre-trained Modelscited by 1
  15. 2025AISTAT lab system for DCASE2025 Task6: Language-based audio retrievalcited by 1
  16. 2025AMAuT: A Flexible and Efficient Multiview Audio Transformer Framework Trained from Scratchcited by 1
  17. 2026Multi-layer attentive probing improves transfer of audio representations for bioacousticscited by 1
  18. 2026Masked Autoencoders with Limited Data: Does It Work? A Fine-Grained Bioacoustics Case Studycited by 1
  19. 2026AudioMosaic: Contrastive Masked Audio Representation Learningcited by 1
  20. 2026From Physics to Representation: Audio Learning with Synthetic Pre-training via Procedural Generationcited by 1
  21. 2026Can We Predict Anomaly Detection Performance from Embedding-Space Geometry?cited by 1
  22. 2025Classifying Shelf Life Quality of Pineapples by Combining Audio and Visual Featurescited by 0
  23. 2025ECHO: Frequency-aware Hierarchical Encoding for Variable-length Signalscited by 0
  24. 2025Improving Anomalous Sound Detection with Attribute-aware Representation from Domain-adaptive Pre-trainingcited by 0
  25. 2025SAM: A Mamba-2 State-Space Audio-Language Modelcited by 0
  26. 2025Representation-Based Data Quality Audits for Audiocited by 0
  27. 2025AaSP: Aliasing-aware Self-Supervised Pre-Training for Audio Spectrogram Transformerscited by 0
  28. 2025BEAT2AASIST model with layer fusion for ESDD 2026 Challengecited by 0
  29. 2026Representation-Regularized Convolutional Audio Transformer for Audio Understandingcited by 0
  30. 2026TLDiffGAN: A Latent Diffusion-GAN Framework with Temporal Information Fusion for Anomalous Sound Detectioncited by 0
  31. 2026BAT: Better Audio Transformer Guided by Convex Gated Probingcited by 0
  32. 2026Sub-Band Spectral Matching with Localized Score Aggregation for Robust Anomalous Sound Detectioncited by 0
  33. 2026Tadabur: A Large-Scale Quran Audio Datasetcited by 0
  34. 2026USAD 2.0: Scaling Representation Distillation for Universal Audio Understandingcited by 0
  35. 2026Training-Free Model Selection and Domain-Aware Score Calibration for First-Shot Anomalous Sound Detectioncited by 0
  36. 2026Can Tokens Compete? Token Representations against Supervised CNN Backbones for BirdCLEF+ 2026cited by 0
  37. 2026Listening Forward: Next Patch Embedding Prediction Enables Scalable Audio Learnerscited by 0
  38. 2026Efficient Passive Acoustic Monitoring of Killer Whales Using a Two-Stage Detection and Ecotype Classification Cascadecited by 0

50 most cited of 52 All 52 (JSON)