mlsanityData to 14 Oct 2026

Self-supervised Audio Teacher-Student Transformer for Both Clip-level and Frame-level Tasks

arXiv 2306.04186 ↗First posted 7 Jun 2023eess.AS cs.LGcited by 25

On the map: Knowledge distillation → Speech and audio distillation

Abstract (arXiv)

Self-supervised learning (SSL) has emerged as a popular approach for learning audio representations. One goal of audio self-supervised pre-training is to transfer knowledge to downstream audio tasks, generally including clip-level and frame-level tasks. While frame-level tasks are important for fine-grained acoustic scene/event understanding, prior studies primarily evaluate on clip-level downstream tasks. In order to tackle both clip-level and frame-level tasks, this paper proposes Audio Teacher-Student Transformer (ATST), with a clip-level version (named ATST-Clip) and a frame-level version (named ATST-Frame), responsible for learning clip-level and frame-level representations, respectively. Both methods use a Transformer encoder and a teacher-student training scheme. We have carefully designed the view creation strategy for ATST-Clip and ATST-Frame. Specifically, ATST-Clip uses segment-wise data augmentations, and ATST-Frame integrates frame-wise data augmentations and masking. Experimental results show that our ATST-Frame model obtains state-of-the-art (SOTA) performances on most of the clip-level and frame-level downstream tasks. Especially, it outperforms other models by a large margin on the frame-level sound event detection task. In addition, the performance can be further improved by combining the two models through knowledge distillation. Our code is available online.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation. Builds on 50 of its 63 references that are in the map. First posted on arXiv 7 Jun 2023.

Builds on

50 of 63 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022ATST: Audio Representation Learning with Teacher-Student Transformer26 Apr
  2. 2022CMKD: CNN/Transformer-Based Cross-Model Knowledge Distillation for Audio Classification13 Mar

Other references · 48

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  8. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  9. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  10. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  11. 2014Going Deeper with Convolutionscited by 4,241
  12. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
36 more
  1. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  2. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  4. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 752
  5. 2017VoxCeleb: a large-scale speaker identification datasetcited by 732
  6. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  7. 2021AST: Audio Spectrogram Transformercited by 643
  8. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 574
  9. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 388
  10. 2020FSD50K: An Open Dataset of Human-Labeled Sound Eventscited by 342
  11. 2019Learning Deep Transformer Models for Machine Translationcited by 309
  12. 2022Masked Autoencoders that Listencited by 273
  13. 2021Efficient Training of Audio Transformers with Patchoutcited by 226
  14. 2017Neural Audio Synthesis of Musical Notes with WaveNet Autoencoderscited by 225
  15. 2019Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoderscited by 198
  16. 2020TERA: Self-Supervised Learning of Transformer Encoder Representation for Speechcited by 188
  17. 2022HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detectioncited by 176
  18. 2020Contrastive Learning of General-Purpose Audio Representationscited by 169
  19. 2021SSAST: Self-Supervised Audio Spectrogram Transformercited by 169
  20. 2021BYOL for Audio: Self-Supervised Learning for General-Purpose Audio Representationcited by 106
  21. 2019Learning Problem-agnostic Speech Representations from Multiple Self-supervised Taskscited by 87
  22. 2020Towards Learning a Universal Non-Semantic Representation of Speechcited by 84
  23. 2021PSLA: Improving Audio Tagging with Pretraining, Sampling, Labeling, and Aggregationcited by 82
  24. 2022MAE-AST: Masked Autoencoding Audio Spectrogram Transformercited by 82
  25. 2022HEAR: Holistic Evaluation of Audio Representationscited by 78
  26. 2022Audio Self-supervised Learning: A Surveycited by 65
  27. 2017Learning from Between-class Examples for Deep Sound Recognitioncited by 54
  28. 2022Masked Spectrogram Modeling using Masked Autoencoders for Learning General-purpose Audio Representationcited by 52
  29. 2021Towards Learning Universal Audio Representationscited by 51
  30. 2022Masked Spectrogram Prediction For Self-Supervised Audio Pre-Trainingcited by 51
  31. 2022BYOL for Audio: Exploring Pre-trained General-purpose Audio Representationscited by 36
  32. 2020Unsupervised Contrastive Learning of Sound Event Representationscited by 32
  33. 2019A Framework for the Robust Evaluation of Sound Event Detectioncited by 29
  34. 2022Masked Modeling Duo: Learning Representations by Encouraging Both Networks to Model the Inputcited by 23
  35. 2021Conformer-Based Self-Supervised Learning for Non-Speech Audio Taskscited by 22
  36. 2022ASiT: Local-Global Audio Spectrogram vIsion Transformer for Event Classificationcited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 13; 2026: 9.

25 papers.

Sort:

By area:

  1. 2024EAT: Self-Supervised Pre-Training with Efficient Audio Transformercited by 53
  2. 2025HunyuanVideo-Foley: Multimodal Diffusion with Representation Alignment for High-Fidelity Foley Audio Generationcited by 46
  3. 2025Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learningcited by 13
  4. 2025Token Pruning in Audio Transformers: Optimizing Performance and Decoding Patch Importancecited by 12
  5. 2025TACOS: Temporally-aligned Audio CaptiOnS for Language-Audio Pretrainingcited by 9
  6. 2025USAD: Universal Speech and Audio Representation via Distillationcited by 7
  7. 2025Unmute the Patch Tokens: Rethinking Probing in Multi-Label Audio Classificationcited by 7
  8. 2025MATPAC++: Enhanced Masked Latent Prediction for Self-Supervised Audio Representation Learningcited by 6
  9. 2025Masked Latent Prediction and Classification for Self-Supervised Audio Representation Learningcited by 4
  10. 2025FlexSED: Towards Open-Vocabulary Sound Event Detectioncited by 4
  11. 2024Estimated Audio-Caption Correspondences Improve Language-Based Audio Retrievalcited by 2
  12. 2024Towards reliable respiratory disease diagnosis based on cough sounds and vision transformerscited by 1
13 more
  1. 2026USAD 2.0: Scaling Representation Distillation for Universal Audio Understandingcited by 1
  2. 2026From Physics to Representation: Audio Learning with Synthetic Pre-training via Procedural Generationcited by 1
  3. 2026RealDESED: A Real-World Domestic Sound Event Detection Benchmarkcited by 1
  4. 2025Exploring Performance-Complexity Trade-Offs in Sound Event Detection Modelscited by 0
  5. 2025Revisiting SSL for sound event detection: complementary fusion and adaptive post-processingcited by 0
  6. 2025Audio-3DVG: Unified Audio -- Point Cloud Fusion for 3D Visual Groundingcited by 0
  7. 2025AaSP: Aliasing-aware Self-Supervised Pre-Training for Audio Spectrogram Transformerscited by 0
  8. 2026Representation-Regularized Convolutional Audio Transformer for Audio Understandingcited by 0
  9. 2026TLDiffGAN: A Latent Diffusion-GAN Framework with Temporal Information Fusion for Anomalous Sound Detectioncited by 0
  10. 2026ImmersiveTTS: Environment-Aware Text-to-Speech with Multimodal Diffusion Transformer and Domain-Specific Representation Alignmentcited by 0
  11. 2026Semi-Supervised Sound Event Detection with Conditional Mixup and Embedding-Level Contrastive Losscited by 0
  12. 2026Augmenting Large Audio-Language Models with Frame-Level Grounding for Fine-Grained Temporal Perceptioncited by 0
  13. 2026Bearings: Self-Supervised Soundfield Embeddings from First-Order Ambisonicscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Leveraging Self-supervised Audio Representations for Data-Efficient Acoustic Scene Classification
  2. 2025Chunk Based Speech Pre-training with High Resolution Finite Scalar Quantization
  3. 2026S-SONDO: Self-Supervised Knowledge Distillation for General Audio Foundation Models
  4. 2024DASS: Distilled Audio State Space Models Are Stronger and More Duration-Scalable Learners
  5. 2024Audio Mamba: Selective State Spaces for Self-Supervised Audio Representations
  6. 2023Incremental Learning of Acoustic Scenes and Sound Events
  7. 2023Recycle-and-Distill: Universal Compression Strategy for Transformer-based Speech SSL Models with Attention Map Reusing and Masking Distillation
  8. 2025An overview of neural architectures for self-supervised audio representation learning from masked spectrograms