# On the Effect of Data-Augmentation on Local Embedding Properties in the Contrastive Learning of Music Audio Representations

arXiv: https://arxiv.org/abs/2401.08889 · HTML: https://mlsanity.com/paper/2401.08889/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 41 references that are in the map.
- First posted on arXiv 17 Jan 2024.

## Builds on (most cited first, 23 of 23)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders](https://mlsanity.com/paper/1704.01279/) (2017)
- [MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training](https://mlsanity.com/paper/2306.00107/) (2023)
- [Contrastive Learning of General-Purpose Audio Representations](https://mlsanity.com/paper/2010.10915/) (2020)
- [Contrastive Learning of Musical Representations](https://mlsanity.com/paper/2103.09410/) (2021)
- [Data Augmenting Contrastive Learning of Speech Representations in the Time Domain](https://mlsanity.com/paper/2007.00991/) (2020)
- [Towards Learning Universal Audio Representations](https://mlsanity.com/paper/2111.12124/) (2021)
- [Unsupervised Learning of Semantic Audio Representations](https://mlsanity.com/paper/1711.02209/) (2017)
- [Multi-Format Contrastive Learning of Audio Representations](https://mlsanity.com/paper/2103.06508/) (2021)
- [BYOL for Audio: Exploring Pre-trained General-purpose Audio Representations](https://mlsanity.com/paper/2204.07402/) (2022)
- [Supervised and Unsupervised Learning of Audio Representations for Music Understanding](https://mlsanity.com/paper/2210.03799/) (2022)
- [Unsupervised Contrastive Learning of Sound Event Representations](https://mlsanity.com/paper/2011.07616/) (2020)
- [Disentangled Multidimensional Metric Learning for Music Similarity](https://mlsanity.com/paper/2008.03720/) (2020)
- [Unsupervised Learning of Deep Features for Music Segmentation](https://mlsanity.com/paper/2108.12955/) (2021)
- [Contrastive Learning with Positive-Negative Frame Mask for Music Representation](https://mlsanity.com/paper/2203.09129/) (2022)
- [Metric Learning vs Classification for Disentangled Music Representation Learning](https://mlsanity.com/paper/2008.03729/) (2020)
- [S3T: Self-Supervised Pre-training with Swin Transformer for Music Classification](https://mlsanity.com/paper/2202.10139/) (2022)
- [Towards Proper Contrastive Self-supervised Learning Strategies For Music Audio Representation](https://mlsanity.com/paper/2207.04471/) (2022)
- [MAST: Multiscale Audio Spectrogram Transformers](https://mlsanity.com/paper/2211.01515/) (2022)
- [Pre-Training Strategies Using Contrastive Learning and Playlist Information for Music Classification and Similarity](https://mlsanity.com/paper/2304.12257/) (2023)
- [Improving Self-Supervised Learning for Audio Representations by Feature Diversity and Decorrelation](https://mlsanity.com/paper/2303.03717/) (2023)

## Built on by (most cited first, 4 of 4)

- [Foundation Models for Music: A Survey](https://mlsanity.com/paper/2408.14340/) (2024)
- [Toward Fully Self-Supervised Multi-Pitch Estimation](https://mlsanity.com/paper/2402.15569/) (2024)
- [Myna: Masking-Based Contrastive Learning of Musical Representations](https://mlsanity.com/paper/2502.12511/) (2025)
- [Emergent musical properties of a transformer under contrastive self-supervised learning](https://mlsanity.com/paper/2506.23873/) (2025)

## Abstract (arXiv)

Audio embeddings are crucial tools in understanding large catalogs of music. Typically embeddings are evaluated on the basis of the performance they provide in a wide range of downstream tasks, however few studies have investigated the local properties of the embedding spaces themselves which are important in nearest neighbor algorithms, commonly used in music search and recommendation. In this work we show that when learning audio representations on music datasets via contrastive learning, musical properties that are typically homogeneous within a track (e.g., key and tempo) are reflected in the locality of neighborhoods in the resulting embedding space. By applying appropriate data augmentation strategies, localisation of such properties can not only be reduced but the localisation of other attributes is increased. For example, locality of features such as pitch and tempo that are less relevant to non-expert listeners, may be mitigated while improving the locality of more salient features such as genre and mood, achieving state-of-the-art performance in nearest neighbor retrieval accuracy. Similarly, we show that the optimal selection of data augmentation strategies for contrastive learning of music audio embeddings is dependent on the downstream task, highlighting this as an important embedding design decision.