# A Convolutional-Attentional Neural Framework for Structure-Aware Performance-Score Synchronization

arXiv: https://arxiv.org/abs/2204.08822 · HTML: https://mlsanity.com/paper/2204.08822/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 45 references that are in the map.
- First posted on arXiv 19 Apr 2022.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Attention U-Net: Learning Where to Look for the Pancreas](https://mlsanity.com/paper/1804.03999/) (2018)
- [Stand-Alone Self-Attention in Vision Models](https://mlsanity.com/paper/1906.05909/) (2019)
- [Soft-DTW: a Differentiable Loss Function for Time-Series](https://mlsanity.com/paper/1703.01541/) (2017)
- [Differentiable Divergences Between Time Series](https://mlsanity.com/paper/2010.08354/) (2020)
- [A Neural Multi-sequence Alignment TeCHnique (NeuMATCH)](https://mlsanity.com/paper/1803.00057/) (2018)
- [Learning to Listen, Read, and Follow: Score Following as a Reinforcement Learning Game](https://mlsanity.com/paper/1807.06391/) (2018)
- [Learning Audio - Sheet Music Correspondences for Score Identification and Offline Alignment](https://mlsanity.com/paper/1707.09887/) (2017)
- [Neural Time Warping For Multiple Sequence Alignment](https://mlsanity.com/paper/2006.15753/) (2020)
- [Learning Frame Similarity using Siamese networks for Audio-to-Score Alignment](https://mlsanity.com/paper/2011.07546/) (2020)
- [Structure-Aware Audio-to-Score Alignment using Progressively Dilated Convolutional Neural Networks](https://mlsanity.com/paper/2102.00382/) (2021)
- [Learning to Read and Follow Music in Complete Score Sheet Images](https://mlsanity.com/paper/2007.10736/) (2020)
- [Audio-Conditioned U-Net for Position Estimation in Full Sheet Images](https://mlsanity.com/paper/1910.07254/) (2019)
- [MIDI-Sheet Music Alignment Using Bootleg Score Synthesis](https://mlsanity.com/paper/2004.10345/) (2020)
- [A Hybrid Approach to Audio-to-Score Alignment](https://mlsanity.com/paper/2007.14333/) (2020)

## Built on by (most cited first, 5 of 5)

- [Fast-FNet: Accelerating Transformer Encoder Models via Efficient Fourier Layers](https://mlsanity.com/paper/2209.12816/) (2022)
- [Soft Dynamic Time Warping for Multi-Pitch Estimation and Beyond](https://mlsanity.com/paper/2304.05032/) (2023)
- [Self-Supervised Contrastive Learning for Robust Audio-Sheet Music Retrieval Systems](https://mlsanity.com/paper/2309.12134/) (2023)
- [Towards Context-Aware Neural Performance-Score Synchronisation](https://mlsanity.com/paper/2206.00454/) (2022)
- [MMSD-Net: Towards Multi-modal Stuttering Detection](https://mlsanity.com/paper/2407.11492/) (2024)

## Abstract (arXiv)

Performance-score synchronization is an integral task in signal processing, which entails generating an accurate mapping between an audio recording of a performance and the corresponding musical score. Traditional synchronization methods compute alignment using knowledge-driven and stochastic approaches, and are typically unable to generalize well to different domains and modalities. We present a novel data-driven method for structure-aware performance-score synchronization. We propose a convolutional-attentional architecture trained with a custom loss based on time-series divergence. We conduct experiments for the audio-to-MIDI and audio-to-image alignment tasks pertained to different score modalities. We validate the effectiveness of our method via ablation studies and comparisons with state-of-the-art alignment approaches. We demonstrate that our approach outperforms previous synchronization methods for a variety of test settings across score modalities and acoustic conditions. Our method is also robust to structural differences between the performance and score sequences, which is a common limitation of standard alignment approaches.