# Equivariant Self-Supervision for Musical Tempo Estimation

arXiv: https://arxiv.org/abs/2209.01478 · HTML: https://mlsanity.com/paper/2209.01478/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 51 references that are in the map.
- First posted on arXiv 3 Sep 2022.

## Builds on (most cited first, 27 of 27)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Deep Clustering for Unsupervised Learning of Visual Features](https://mlsanity.com/paper/1807.05520/) (2018)
- [What Makes for Good Views for Contrastive Learning?](https://mlsanity.com/paper/2005.10243/) (2020)
- [Jukebox: A Generative Model for Music](https://mlsanity.com/paper/2005.00341/) (2020)
- [vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations](https://mlsanity.com/paper/1910.05453/) (2019)
- [Self-supervised Equivariant Attention Mechanism for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2004.04581/) (2020)
- [BYOL for Audio: Self-Supervised Learning for General-Purpose Audio Representation](https://mlsanity.com/paper/2103.06695/) (2021)
- [Contrastive Learning of Musical Representations](https://mlsanity.com/paper/2103.09410/) (2021)
- [Equivariant Contrastive Learning](https://mlsanity.com/paper/2111.00899/) (2021)
- [Towards Learning Universal Audio Representations](https://mlsanity.com/paper/2111.12124/) (2021)
- [Time-Equivariant Contrastive Video Representation Learning](https://mlsanity.com/paper/2112.03624/) (2021)
- [CDPAM: Contrastive learning for perceptual audio similarity](https://mlsanity.com/paper/2102.05109/) (2021)
- [Speech SIMCLR: Combining Contrastive and Reconstruction Objective for Self-supervised Speech Representation Learning](https://mlsanity.com/paper/2010.13991/) (2020)
- [SPICE: Self-supervised Pitch Estimation](https://mlsanity.com/paper/1910.11664/) (2019)
- [Conformer-Based Self-Supervised Learning for Non-Speech Audio Tasks](https://mlsanity.com/paper/2110.07313/) (2021)
- [Contrastive Learning with Positive-Negative Frame Mask for Music Representation](https://mlsanity.com/paper/2203.09129/) (2022)
- [S3T: Self-Supervised Pre-training with Swin Transformer for Music Classification](https://mlsanity.com/paper/2202.10139/) (2022)
- [Musical Tempo Estimation Using a Multi-scale Network](https://mlsanity.com/paper/2109.01607/) (2021)

## Built on by (most cited first, 9 of 9)

- [PESTO: Real-Time Pitch Estimation with Self-supervised Transposition-equivariant Objective](https://mlsanity.com/paper/2508.01488/) (2025)
- [Tempo vs. Pitch: understanding self-supervised tempo estimation](https://mlsanity.com/paper/2304.06868/) (2023)
- [Emergent musical properties of a transformer under contrastive self-supervised learning](https://mlsanity.com/paper/2506.23873/) (2025)
- [Tempo estimation as fully self-supervised binary classification](https://mlsanity.com/paper/2401.08891/) (2024)
- [Similar but Faster: Manipulation of Tempo in Music Audio Embeddings for Tempo Prediction and Search](https://mlsanity.com/paper/2401.08902/) (2024)
- [Translation-Equivariant Self-Supervised Learning for Pitch Estimation with Optimal Transport](https://mlsanity.com/paper/2508.01493/) (2025)
- [MIDI-RAE-JEPA: Hierarchical Representation Learning and Generation for Symbolic Music](https://mlsanity.com/paper/2607.14537/) (2026)
- [Equivariant Music Transformer](https://mlsanity.com/paper/2608.03920/) (2026)
- [CoJEPA: Combining Contrastive Learning and JEPA for Global-Local Music Representations](https://mlsanity.com/paper/2608.30974/) (2026)

## Abstract (arXiv)

Self-supervised methods have emerged as a promising avenue for representation learning in the recent years since they alleviate the need for labeled datasets, which are scarce and expensive to acquire. Contrastive methods are a popular choice for self-supervision in the audio domain, and typically provide a learning signal by forcing the model to be invariant to some transformations of the input. These methods, however, require measures such as negative sampling or some form of regularisation to be taken to prevent the model from collapsing on trivial solutions. In this work, instead of invariance, we propose to use equivariance as a self-supervision signal to learn audio tempo representations from unlabelled data. We derive a simple loss function that prevents the network from collapsing on a trivial solution during training, without requiring any form of regularisation or negative sampling. Our experiments show that it is possible to learn meaningful representations for tempo estimation by solely relying on equivariant self-supervision, achieving performance comparable with supervised methods on several benchmarks. As an added benefit, our method only requires moderate compute resources and therefore remains accessible to a wide research community.