# MT-SLVR: Multi-Task Self-Supervised Learning for Transformation In(Variant) Representations

arXiv: https://arxiv.org/abs/2305.17191 · HTML: https://mlsanity.com/paper/2305.17191/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 35 references that are in the map.
- First posted on arXiv 29 May 2023.

## Builds on (most cited first, 27 of 27)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition](https://mlsanity.com/paper/1804.03209/) (2018)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [Do Better ImageNet Models Transfer Better?](https://mlsanity.com/paper/1805.08974/) (2018)
- [Learning multiple visual domains with residual adapters](https://mlsanity.com/paper/1705.08045/) (2017)
- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders](https://mlsanity.com/paper/1704.01279/) (2017)
- [Contrastive Learning of General-Purpose Audio Representations](https://mlsanity.com/paper/2010.10915/) (2020)
- [Self-Supervised Representation Learning: Introduction, Advances and Challenges](https://mlsanity.com/paper/2110.09327/) (2021)
- [Multi-task self-supervised learning for Robust Speech Recognition](https://mlsanity.com/paper/2001.09239/) (2020)
- [Audio Self-supervised Learning: A Survey](https://mlsanity.com/paper/2203.01205/) (2022)
- [Improving Transferability of Representations via Augmentation-Aware Self-Supervision](https://mlsanity.com/paper/2111.09613/) (2021)
- [Unsupervised Contrastive Learning of Sound Event Representations](https://mlsanity.com/paper/2011.07616/) (2020)
- [CLAR: Contrastive Learning of Auditory Representations](https://mlsanity.com/paper/2010.09542/) (2020)
- [General-purpose Tagging of Freesound Audio with AudioSet Labels: Task Description, Dataset, and Baseline](https://mlsanity.com/paper/1807.09902/) (2018)
- [MetaAudio: A Few-Shot Audio Classification Benchmark](https://mlsanity.com/paper/2204.02121/) (2022)
- [Why Do Self-Supervised Models Transfer? Investigating the Impact of Invariance on Downstream Tasks](https://mlsanity.com/paper/2111.11398/) (2021)
- [Learning to match transient sound events using attentional similarity for few-shot sound recognition](https://mlsanity.com/paper/1812.01269/) (2018)
- [Who calls the shots? Rethinking Few-Shot Learning for Audio](https://mlsanity.com/paper/2110.09600/) (2021)
- [Amortised Invariance Learning for Contrastive Self-Supervision](https://mlsanity.com/paper/2302.12712/) (2023)

## Built on by (most cited first, 3 of 3)

- [On the Transferability of Large-Scale Self-Supervision to Few-Shot Audio Classification](https://mlsanity.com/paper/2402.01274/) (2024)
- [Self-supervised Learning for Acoustic Few-Shot Classification](https://mlsanity.com/paper/2409.09647/) (2024)
- [uaMix-MAE: Efficient Tuning of Pretrained Audio Transformers with Unsupervised Audio Mixtures](https://mlsanity.com/paper/2403.09579/) (2024)

## Abstract (arXiv)

Contrastive self-supervised learning has gained attention for its ability to create high-quality representations from large unlabelled data sets. A key reason that these powerful features enable data-efficient learning of downstream tasks is that they provide augmentation invariance, which is often a useful inductive bias. However, the amount and type of invariances preferred is not known apriori, and varies across different downstream tasks. We therefore propose a multi-task self-supervised framework (MT-SLVR) that learns both variant and invariant features in a parameter-efficient manner. Our multi-task representation provides a strong and flexible feature that benefits diverse downstream tasks. We evaluate our approach on few-shot classification tasks drawn from a variety of audio domains and demonstrate improved classification performance on all of them