# Self-Supervised Beat Tracking in Musical Signals with Polyphonic Contrastive Learning

arXiv: https://arxiv.org/abs/2201.01771 · HTML: https://mlsanity.com/paper/2201.01771/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 68 references that are in the map.
- First posted on arXiv 5 Jan 2022.

## Builds on (most cited first, 29 of 29)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [Look, Listen and Learn](https://mlsanity.com/paper/1705.08168/) (2017)
- [The Sound of Pixels](https://mlsanity.com/paper/1804.03160/) (2018)
- [FMA: A Dataset For Music Analysis](https://mlsanity.com/paper/1612.01840/) (2016)
- [What makes ImageNet good for transfer learning?](https://mlsanity.com/paper/1608.08614/) (2016)
- [DDSP: Differentiable Digital Signal Processing](https://mlsanity.com/paper/2001.04643/) (2020)
- [Contrastive Learning of General-Purpose Audio Representations](https://mlsanity.com/paper/2010.10915/) (2020)
- [Wave-U-Net: A Multi-Scale Neural Network for End-to-End Audio Source Separation](https://mlsanity.com/paper/1806.03185/) (2018)
- [Contrastive Learning for Label-Efficient Semantic Segmentation](https://mlsanity.com/paper/2012.06985/) (2020)
- [Contrastive Learning of Musical Representations](https://mlsanity.com/paper/2103.09410/) (2021)
- [madmom: a new Python Audio and Music Signal Processing Library](https://mlsanity.com/paper/1605.07008/) (2016)
- [Unsupervised Learning of Semantic Audio Representations](https://mlsanity.com/paper/1711.02209/) (2017)
- [Multi-Format Contrastive Learning of Audio Representations](https://mlsanity.com/paper/2103.06508/) (2021)
- [SPICE: Self-supervised Pitch Estimation](https://mlsanity.com/paper/1910.11664/) (2019)
- [End-to-end music source separation: is it possible in the waveform domain?](https://mlsanity.com/paper/1810.12187/) (2018)
- [Multi-Task Self-Supervised Pre-Training for Music Classification](https://mlsanity.com/paper/2102.03229/) (2021)
- [Self-supervised audio representation learning for mobile devices](https://mlsanity.com/paper/1905.11796/) (2019)
- [Comparison of Time-Frequency Representations for Environmental Sound Classification using Convolutional Neural Networks](https://mlsanity.com/paper/1706.07156/) (2017)
- [Self-Supervised Learning of Audio Representations from Permutations with Differentiable Ranking](https://mlsanity.com/paper/2103.09879/) (2021)
- [Metric Learning vs Classification for Disentangled Music Representation Learning](https://mlsanity.com/paper/2008.03729/) (2020)
- [Learning a Joint Embedding Space of Monophonic and Mixed Music Signals for Singing Voice](https://mlsanity.com/paper/1906.11139/) (2019)

## Abstract (arXiv)

Annotating musical beats is a very long and tedious process. In order to combat this problem, we present a new self-supervised learning pretext task for beat tracking and downbeat estimation. This task makes use of Spleeter, an audio source separation model, to separate a song's drums from the rest of its signal. The first set of signals are used as positives, and by extension negatives, for contrastive learning pre-training. The drum-less signals, on the other hand, are used as anchors. When pre-training a fully-convolutional and recurrent model using this pretext task, an onset function is learned. In some cases, this function is found to be mapped to periodic elements in a song. We find that pre-trained models outperform randomly initialized models when a beat tracking training set is extremely small (less than 10 examples). When this is not the case, pre-training leads to a learning speed-up that causes the model to overfit to the training set. More generally, this work defines new perspectives in the realm of musical self-supervised learning. It is notably one of the first works to use audio source separation as a fundamental component of self-supervision.