mlsanityData to 14 Oct 2026

ATST: Audio Representation Learning with Teacher-Student Transformer

arXiv 2204.12076 ↗First posted 26 Apr 2022eess.AS cs.AI cs.SDcited by 13

On the map: Knowledge distillation → Speech and audio distillation

Abstract (arXiv)

Self-supervised learning (SSL) learns knowledge from a large amount of unlabeled data, and then transfers the knowledge to a specific problem with a limited number of labeled data. SSL has achieved promising results in various domains. This work addresses the problem of segment-level general audio SSL, and proposes a new transformer-based teacher-student SSL model, named ATST. A transformer encoder is developed on a recently emerged teacher-student baseline scheme, which largely improves the modeling capability of pre-training. In addition, a new strategy for positive pair creation is designed to fully leverage the capability of transformer. Extensive experiments have been conducted, and the proposed model achieves the new state-of-the-art results on almost all of the downstream tasks.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation. Builds on 21 of its 25 references that are in the map. First posted on arXiv 26 Apr 2022.

Builds on

21 of 25 references.

References · 21

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  5. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  6. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  7. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  8. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  9. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 752
  10. 2017VoxCeleb: a large-scale speaker identification datasetcited by 732
  11. 2021AST: Audio Spectrogram Transformercited by 643
  12. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 574
9 more
  1. 2017Neural Audio Synthesis of Musical Notes with WaveNet Autoencoderscited by 225
  2. 2019Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoderscited by 198
  3. 2020TERA: Self-Supervised Learning of Transformer Encoder Representation for Speechcited by 188
  4. 2020Contrastive Learning of General-Purpose Audio Representationscited by 169
  5. 2021SSAST: Self-Supervised Audio Spectrogram Transformercited by 169
  6. 2021BYOL for Audio: Self-Supervised Learning for General-Purpose Audio Representationcited by 106
  7. 2020Towards Learning a Universal Non-Semantic Representation of Speechcited by 84
  8. 2020Unsupervised Contrastive Learning of Sound Event Representationscited by 32
  9. 2021Conformer-Based Self-Supervised Learning for Non-Speech Audio Taskscited by 22

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 1; 2024: 3; 2025: 2; 2026: 5.

13 papers.

Sort:

By area:

  1. 2024EAT: Self-Supervised Pre-Training with Efficient Audio Transformercited by 53
  2. 2023Self-supervised Audio Teacher-Student Transformer for Both Clip-level and Frame-level Taskscited by 25
  3. 2022Masked Modeling Duo: Learning Representations by Encouraging Both Networks to Model the Inputcited by 23
  4. 2022ASiT: Local-Global Audio Spectrogram vIsion Transformer for Event Classificationcited by 13
  5. 2024Stem-JEPA: A Joint-Embedding Predictive Architecture for Musical Stem Compatibility Estimationcited by 8
  6. 2024Investigating Design Choices in Joint-Embedding Predictive Architectures for General Audio Representation Learningcited by 4
  7. 2025Revisiting Audio-language Pretraining for Learning General-purpose Audio Representationcited by 4
  8. 2026Unlocking Strong Supervision: A Data-Centric Study of General-Purpose Audio Pre-Training Methodscited by 1
  9. 2025Hierarchical Label Propagation: A Model-Size-Dependent Performance Booster for AudioSet Taggingcited by 0
  10. 2026Semi-Supervised Sound Event Detection with Conditional Mixup and Embedding-Level Contrastive Losscited by 0
  11. 2026TriA Pipeline: A Large-Scale Automatic Audio Annotation Pipeline For Audio Classification In Specific Scenarioscited by 0
  12. 2026CASA: Content-Acoustic Speaking Assessment with Speech Encoder and Large Language Modelcited by 0
1 more
  1. 2026Alignment Is All You Need: Instruction-Free Training for General Audio-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Leveraging Self-supervised Audio Representations for Data-Efficient Acoustic Scene Classification
  2. 2023Recycle-and-Distill: Universal Compression Strategy for Transformer-based Speech SSL Models with Attention Map Reusing and Masking Distillation
  3. 2026S-SONDO: Self-Supervised Knowledge Distillation for General Audio Foundation Models
  4. 2024DASS: Distilled Audio State Space Models Are Stronger and More Duration-Scalable Learners
  5. 2021Exploiting Large-scale Teacher-Student Training for On-device Acoustic Models
  6. 2026Multi-Teacher Distillation for Cross-Domain Streaming Electrolaryngeal Speech Encoding ↗
  7. 2022data2vec-aqc: Search for the right Teaching Assistant in the Teacher-Student training setup
  8. 2022Ultra Fast Speech Separation Model with Teacher Student Learning