# Singer Identity Representation Learning using Self-Supervised Techniques

arXiv: https://arxiv.org/abs/2401.05064 · HTML: https://mlsanity.com/paper/2401.05064/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 51 references that are in the map.
- First posted on arXiv 10 Jan 2024.

## Builds on (most cited first, 29 of 29)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning](https://mlsanity.com/paper/2105.04906/) (2021)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification](https://mlsanity.com/paper/2005.07143/) (2020)
- [Unsupervised Cross-lingual Representation Learning for Speech Recognition](https://mlsanity.com/paper/2006.13979/) (2020)
- [Understanding the Behaviour of Contrastive Loss](https://mlsanity.com/paper/2012.09740/) (2020)
- [Generalized End-to-End Loss for Speaker Verification](https://mlsanity.com/paper/1710.10467/) (2017)
- [Contrastive Learning of General-Purpose Audio Representations](https://mlsanity.com/paper/2010.10915/) (2020)
- [Decoupled Contrastive Learning](https://mlsanity.com/paper/2110.06848/) (2021)
- [In defence of metric learning for speaker recognition](https://mlsanity.com/paper/2003.11982/) (2020)
- [Exploring wav2vec 2.0 on speaker verification and language identification](https://mlsanity.com/paper/2012.06185/) (2020)
- [Contrastive Learning of Musical Representations](https://mlsanity.com/paper/2103.09410/) (2021)
- [Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representations](https://mlsanity.com/paper/2110.14513/) (2021)
- [Pushing the limits of raw waveform speaker recognition](https://mlsanity.com/paper/2203.08488/) (2022)
- [BYOL for Audio: Exploring Pre-trained General-purpose Audio Representations](https://mlsanity.com/paper/2204.07402/) (2022)
- [Self-supervised Text-independent Speaker Verification using Prototypical Momentum Contrastive Learning](https://mlsanity.com/paper/2012.07178/) (2020)
- [CLAR: Contrastive Learning of Auditory Representations](https://mlsanity.com/paper/2010.09542/) (2020)
- [The ins and outs of speaker recognition: lessons from VoxSRC 2020](https://mlsanity.com/paper/2010.15809/) (2020)
- [Self-Supervised Speaker Verification with Simple Siamese Network and Self-Supervised Regularization](https://mlsanity.com/paper/2112.04459/) (2021)
- [Label-Efficient Self-Supervised Speaker Verification With Information Maximization and Contrastive Learning](https://mlsanity.com/paper/2207.05506/) (2022)
- [SampleMatch: Drum Sample Retrieval by Musical Context](https://mlsanity.com/paper/2208.01141/) (2022)
- [MR-SVS: Singing Voice Synthesis with Multi-Reference Encoder](https://mlsanity.com/paper/2201.03864/) (2022)

## Built on by (most cited first, 6 of 6)

- [Foundation Models for Music: A Survey](https://mlsanity.com/paper/2408.14340/) (2024)
- [Toward Fully Self-Supervised Multi-Pitch Estimation](https://mlsanity.com/paper/2402.15569/) (2024)
- [From Real to Cloned Singer Identification](https://mlsanity.com/paper/2407.08647/) (2024)
- [Neuron Platonic Intrinsic Representation From Dynamics Using Contrastive Learning](https://mlsanity.com/paper/2502.10425/) (2025)
- [Hybrid Losses for Hierarchical Embedding Learning](https://mlsanity.com/paper/2501.12796/) (2025)
- [Music Restoration via Latent Operator Optimization and Diffusion Model Priors](https://mlsanity.com/paper/2608.01972/) (2026)

## Abstract (arXiv)

Significant strides have been made in creating voice identity representations using speech data. However, the same level of progress has not been achieved for singing voices. To bridge this gap, we suggest a framework for training singer identity encoders to extract representations suitable for various singing-related tasks, such as singing voice similarity and synthesis. We explore different self-supervised learning techniques on a large collection of isolated vocal tracks and apply data augmentations during training to ensure that the representations are invariant to pitch and content variations. We evaluate the quality of the resulting representations on singer similarity and identification tasks across multiple datasets, with a particular emphasis on out-of-domain generalization. Our proposed framework produces high-quality embeddings that outperform both speaker verification and wav2vec 2.0 pre-trained baselines on singing voice while operating at 44.1 kHz. We release our code and trained models to facilitate further research on singing voice and related areas.