mlsanityData to 7 Oct 2026

BAT: Better Audio Transformer Guided by Convex Gated Probing

arXiv 2602.16305 ↗First posted 18 Feb 2026cs.SD cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Probing is widely adopted in computer vision to faithfully evaluate self-supervised learning (SSL) embeddings, as finetuning may misrepresent their inherent quality. In contrast, audio SSL models still rely on finetuning because simple probing fails to unlock their full potential and alters their rankings when competing on AudioSet. Hence, a robust and efficient probing mechanism is required to guide the trajectory of audio SSL towards reliable and reproducible methods. We introduce Convex Gated Probing (CGP), a prototype-based method that significantly closes the gap between finetuning and probing in audio. CGP efficiently utilizes all frozen layers via a gating mechanism and exposes the location of latent task-relevant information. Guided by CGP as a reliable post-hoc evaluation probe, we rework the entire SSL pipeline of current best performing audio models that use legacy implementations of prior SSL methods. By refining data preprocessing, model architecture, and pretraining recipe, we introduce Better Audio Transformer (BAT), and establish new SOTA on audio benchmarks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 45 of its 61 references that are in the map. First posted on arXiv 18 Feb 2026.

Builds on

45 of 61 references.

References · 45

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  5. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  7. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  8. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  9. 2015Continuous control with deep reinforcement learningcited by 3,279
  10. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  11. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  12. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
33 more
  1. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  2. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  3. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  4. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  5. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 741
  6. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
  7. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  8. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  9. 2020Asymmetric Loss For Multi-Label Classificationcited by 379
  10. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 378
  11. 2022Masked Siamese Networks for Label-Efficient Learningcited by 301
  12. 2022BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizerscited by 298
  13. 2020Whitening for Self-Supervised Representation Learningcited by 278
  14. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 266
  15. 2022Surgical Fine-Tuning Improves Adaptation to Distribution Shiftscited by 205
  16. 2021SSAST: Self-Supervised Audio Spectrogram Transformercited by 166
  17. 2022Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Languagecited by 108
  18. 2021BYOL for Audio: Self-Supervised Learning for General-Purpose Audio Representationcited by 106
  19. 2022Head2Toe: Utilizing Intermediate Representations for Better Transfer Learningcited by 83
  20. 2022HEAR: Holistic Evaluation of Audio Representationscited by 77
  21. 2022Visual Query Tuning: Towards Effective Usage of Intermediate Representations for Parameter and Memory Efficient Transfer Learningcited by 57
  22. 2022Masked Spectrogram Modeling using Masked Autoencoders for Learning General-purpose Audio Representationcited by 52
  23. 2024EAT: Self-Supervised Pre-Training with Efficient Audio Transformercited by 52
  24. 2022BEANS: The Benchmark of Animal Soundscited by 37
  25. 2024BirdSet: A Large-Scale Dataset for Audio Classification in Avian Bioacousticscited by 36
  26. 2025Can Masked Autoencoders Also Listen to Birds?cited by 26
  27. 2025Cluster and Predict Latent Patches for Improved Masked Image Modelingcited by 17
  28. 2025SSLAM: Enhancing Self-Supervised Models with Audio Mixtures for Polyphonic Soundscapescited by 16
  29. 2025Attention, Please! Revisiting Attentive Probing Through the Lens of Efficiencycited by 11
  30. 2024Towards Deep Active Learning in Avian Bioacousticscited by 10
  31. 2025Unmute the Patch Tokens: Rethinking Probing in Multi-Label Audio Classificationcited by 7
  32. 2024Not Only the Last-Layer Features for Spurious Correlations: All Layer Deep Feature Reweightingcited by 3
  33. 2025Data-Efficient Self-Supervised Algorithms for Fine-Grained Birdsong Analysiscited by 1

Built on by

Not cited yet.