# Learning to Listen, Read, and Follow: Score Following as a Reinforcement Learning Game

arXiv: https://arxiv.org/abs/1807.06391 · HTML: https://mlsanity.com/paper/1807.06391/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Jul 2018.

## Built on by (most cited first, 7 of 7)

- [A Survey on Deep Reinforcement Learning for Audio-Based Applications](https://mlsanity.com/paper/2101.00240/) (2021)
- [Structure-Aware Audio-to-Score Alignment using Progressively Dilated Convolutional Neural Networks](https://mlsanity.com/paper/2102.00382/) (2021)
- [A Convolutional-Attentional Neural Framework for Structure-Aware Performance-Score Synchronization](https://mlsanity.com/paper/2204.08822/) (2022)
- [Online Symbolic Music Alignment with Offline Reinforcement Learning](https://mlsanity.com/paper/2401.00466/) (2023)
- [Multi-modal Conditional Bounding Box Regression for Music Score Following](https://mlsanity.com/paper/2105.04309/) (2021)
- [TheGlueNote: Learned Representations for Robust and Flexible Note Alignment](https://mlsanity.com/paper/2408.04309/) (2024)
- [Proceedings of the 2nd International Workshop on Reading Music Systems](https://mlsanity.com/paper/2212.00380/) (2022)

## Abstract (arXiv)

Score following is the process of tracking a musical performance (audio) with respect to a known symbolic representation (a score). We start this paper by formulating score following as a multimodal Markov Decision Process, the mathematical foundation for sequential decision making. Given this formal definition, we address the score following task with state-of-the-art deep reinforcement learning (RL) algorithms such as synchronous advantage actor critic (A2C). In particular, we design multimodal RL agents that simultaneously learn to listen to music, read the scores from images of sheet music, and follow the audio along in the sheet, in an end-to-end fashion. All this behavior is learned entirely from scratch, based on a weak and potentially delayed reward signal that indicates to the agent how close it is to the correct position in the score. Besides discussing the theoretical advantages of this learning paradigm, we show in experiments that it is in fact superior compared to previously proposed methods for score following in raw sheet music images.