# Passage Summarization with Recurrent Models for Audio-Sheet Music Retrieval

arXiv: https://arxiv.org/abs/2309.12111 · HTML: https://mlsanity.com/paper/2309.12111/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 31 references that are in the map.
- First posted on arXiv 21 Sep 2023.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models](https://mlsanity.com/paper/1411.2539/) (2014)
- [Onsets and Frames: Dual-Objective Piano Transcription](https://mlsanity.com/paper/1710.11153/) (2017)
- [Understanding Optical Music Recognition](https://mlsanity.com/paper/1908.03608/) (2019)
- [Optical Music Recognition with Convolutional Sequence-to-Sequence Models](https://mlsanity.com/paper/1707.04877/) (2017)
- [End-to-End Cross-Modality Retrieval with CCA Projections and Pairwise Ranking Loss](https://mlsanity.com/paper/1705.06979/) (2017)
- [Learning Audio - Sheet Music Correspondences for Score Identification and Offline Alignment](https://mlsanity.com/paper/1707.09887/) (2017)
- [Cross-Modal Music Retrieval and Applications: An Overview of Key Methodologies](https://mlsanity.com/paper/1902.04397/) (2019)
- [Learning Soft-Attention Models for Tempo-invariant Audio-Sheet Music Retrieval](https://mlsanity.com/paper/1906.10996/) (2019)
- [MIDI Passage Retrieval Using Cell Phone Pictures of Sheet Music](https://mlsanity.com/paper/2004.10347/) (2020)
- [Towards Linking the Lakh and IMSLP Datasets](https://mlsanity.com/paper/2004.10391/) (2020)

## Abstract (arXiv)

Many applications of cross-modal music retrieval are related to connecting sheet music images to audio recordings. A typical and recent approach to this is to learn, via deep neural networks, a joint embedding space that correlates short fixed-size snippets of audio and sheet music by means of an appropriate similarity structure. However, two challenges that arise out of this strategy are the requirement of strongly aligned data to train the networks, and the inherent discrepancies of musical content between audio and sheet music snippets caused by local and global tempo differences. In this paper, we address these two shortcomings by designing a cross-modal recurrent network that learns joint embeddings that can summarize longer passages of corresponding audio and sheet music. The benefits of our method are that it only requires weakly aligned audio-sheet music pairs, as well as that the recurrent network handles the non-linearities caused by tempo variations between audio and sheet music. We conduct a number of experiments on synthetic and real piano data and scores, showing that our proposed recurrent method leads to more accurate retrieval in all possible configurations.