# Modeling Beats and Downbeats with a Time-Frequency Transformer

arXiv: https://arxiv.org/abs/2205.14701 · HTML: https://mlsanity.com/paper/2205.14701/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 May 2022.

## Built on by (most cited first, 12 of 12)

- [StemGen: A music generation model that listens](https://mlsanity.com/paper/2312.08723/) (2023)
- [Beat this! Accurate beat tracking without DBN postprocessing](https://mlsanity.com/paper/2407.21658/) (2024)
- [Multitrack Music Transcription with a Time-Frequency Perceiver](https://mlsanity.com/paper/2306.10785/) (2023)
- [Jointist: Joint Learning for Multi-instrument Transcription and Its Applications](https://mlsanity.com/paper/2206.10805/) (2022)
- [A Hybrid Transformer-Mamba Network for Single Image Deraining](https://mlsanity.com/paper/2409.00410/) (2024)
- [Jointist: Simultaneous Improvement of Multi-instrument Transcription and Music Source Separation via Joint Training](https://mlsanity.com/paper/2302.00286/) (2023)
- [MOSA: Music Motion with Semantic Annotation Dataset for Cross-Modal Music Processing](https://mlsanity.com/paper/2406.06375/) (2024)
- [Scaling Up Music Information Retrieval Training with Semi-Supervised Learning](https://mlsanity.com/paper/2310.01353/) (2023)
- [High-Resolution Sustain Pedal Depth Estimation from Piano Audio Across Room Acoustics](https://mlsanity.com/paper/2507.04230/) (2025)
- [Transcribing Rhythmic Patterns of the Guitar Track in Polyphonic Music](https://mlsanity.com/paper/2510.05756/) (2025)
- [AlignBeat: A Latent Variable Model for Multi-Class Beat Tracking from Partially Labeled Data](https://mlsanity.com/paper/2510.14391/) (2025)
- [Explore This! Beat and Downbeat Tracking From a Learned Tatum Grid](https://mlsanity.com/paper/2610.14817/) (2026)

## Abstract (arXiv)

Transformer is a successful deep neural network (DNN) architecture that has shown its versatility not only in natural language processing but also in music information retrieval (MIR). In this paper, we present a novel Transformer-based approach to tackle beat and downbeat tracking. This approach employs SpecTNT (Spectral-Temporal Transformer in Transformer), a variant of Transformer that models both spectral and temporal dimensions of a time-frequency input of music audio. A SpecTNT model uses a stack of blocks, where each consists of two levels of Transformer encoders. The lower-level (or spectral) encoder handles the spectral features and enables the model to pay attention to harmonic components of each frame. Since downbeats indicate bar boundaries and are often accompanied by harmonic changes, this step may help downbeat modeling. The upper-level (or temporal) encoder aggregates useful local spectral information to pay attention to beat/downbeat positions. We also propose an architecture that combines SpecTNT with a state-of-the-art model, Temporal Convolutional Networks (TCN), to further improve the performance. Extensive experiments demonstrate that our approach can significantly outperform TCN in downbeat tracking while maintaining comparable result in beat tracking.