# S3T: Self-Supervised Pre-training with Swin Transformer for Music Classification

arXiv: https://arxiv.org/abs/2202.10139 · HTML: https://mlsanity.com/paper/2202.10139/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Feb 2022.

## Built on by (most cited first, 8 of 8)

- [Audio Self-supervised Learning: A Survey](https://mlsanity.com/paper/2203.01205/) (2022)
- [MuseChat: A Conversational Music Recommendation System for Videos](https://mlsanity.com/paper/2310.06282/) (2023)
- [Equivariant Self-Supervision for Musical Tempo Estimation](https://mlsanity.com/paper/2209.01478/) (2022)
- [On the Effect of Data-Augmentation on Local Embedding Properties in the Contrastive Learning of Music Audio Representations](https://mlsanity.com/paper/2401.08889/) (2024)
- [Myna: Masking-Based Contrastive Learning of Musical Representations](https://mlsanity.com/paper/2502.12511/) (2025)
- [Emergent musical properties of a transformer under contrastive self-supervised learning](https://mlsanity.com/paper/2506.23873/) (2025)
- [Spectrograms Are Sequences of Patches](https://mlsanity.com/paper/2210.15988/) (2022)
- [Quantum Vision Theory Applied to Audio Classification for Deepfake Speech Detection](https://mlsanity.com/paper/2604.08104/) (2026)

## Abstract (arXiv)

In this paper, we propose S3T, a self-supervised pre-training method with Swin Transformer for music classification, aiming to learn meaningful music representations from massive easily accessible unlabeled music data. S3T introduces a momentum-based paradigm, MoCo, with Swin Transformer as its feature extractor to music time-frequency domain. For better music representations learning, S3T contributes a music data augmentation pipeline and two specially designed pre-processors. To our knowledge, S3T is the first method combining the Swin Transformer with a self-supervised learning method for music classification. We evaluate S3T on music genre classification and music tagging tasks with linear classifiers trained on learned representations. Experimental results show that S3T outperforms the previous self-supervised method (CLMR) by 12.5 percents top-1 accuracy and 4.8 percents PR-AUC on two tasks respectively, and also surpasses the task-specific state-of-the-art supervised methods. Besides, S3T shows advances in label efficiency using only 10% labeled data exceeding CLMR on both tasks with 100% labeled data.