# Late multimodal fusion for image and audio music transcription

arXiv: https://arxiv.org/abs/2204.03063 · HTML: https://mlsanity.com/paper/2204.03063/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 39 references that are in the map.
- First posted on arXiv 6 Apr 2022.

## Builds on (most cited first, 7 of 7)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Understanding Optical Music Recognition](https://mlsanity.com/paper/1908.03608/) (2019)
- [Comparison of Decoding Strategies for CTC Acoustic Models](https://mlsanity.com/paper/1708.04469/) (2017)
- [Multimodal music information processing and retrieval: survey and future challenges](https://mlsanity.com/paper/1902.05347/) (2019)
- [Bi-Directional Lattice Recurrent Neural Networks for Confidence Estimation](https://mlsanity.com/paper/1810.13024/) (2018)
- [A holistic approach to polyphonic music transcription with neural networks](https://mlsanity.com/paper/1910.12086/) (2019)
- [Lattice-based lightly-supervised acoustic model training](https://mlsanity.com/paper/1905.13150/) (2019)

## Built on by (most cited first, 3 of 3)

- [Learning to Rebalance Multi-Modal Optimization by Adaptively Masking Subnetworks](https://mlsanity.com/paper/2404.08347/) (2024)
- [Multimodal Classification via Modal-Aware Interactive Enhancement](https://mlsanity.com/paper/2407.04587/) (2024)
- [Trustworthy Enhanced Multi-view Multi-modal Alzheimer's Disease Prediction with Brain-wide Imaging Transcriptomics Data](https://mlsanity.com/paper/2406.14977/) (2024)

## Abstract (arXiv)

Music transcription, which deals with the conversion of music sources into a structured digital format, is a key problem for Music Information Retrieval (MIR). When addressing this challenge in computational terms, the MIR community follows two lines of research: music documents, which is the case of Optical Music Recognition (OMR), or audio recordings, which is the case of Automatic Music Transcription (AMT). The different nature of the aforementioned input data has conditioned these fields to develop modality-specific frameworks. However, their recent definition in terms of sequence labeling tasks leads to a common output representation, which enables research on a combined paradigm. In this respect, multimodal image and audio music transcription comprises the challenge of effectively combining the information conveyed by image and audio modalities. In this work, we explore this question at a late-fusion level: we study four combination approaches in order to merge, for the first time, the hypotheses regarding end-to-end OMR and AMT systems in a lattice-based search space. The results obtained for a series of performance scenarios -- in which the corresponding single-modality models yield different error rates -- showed interesting benefits of these approaches. In addition, two of the four strategies considered significantly improve the corresponding unimodal standard recognition frameworks.