# Multimodal music information processing and retrieval: survey and future challenges

arXiv: https://arxiv.org/abs/1902.05347 · HTML: https://mlsanity.com/paper/1902.05347/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Feb 2019.

## Built on by (most cited first, 13 of 13)

- [What all do audio transformer models hear? Probing Acoustic Representations for Language Delivery and its Structure](https://mlsanity.com/paper/2101.00387/) (2021)
- [LLark: A Multimodal Instruction-Following Language Model for Music](https://mlsanity.com/paper/2310.07160/) (2023)
- [Imbalanced Big Data Oversampling: Taxonomy, Algorithms, Software, Guidelines and Future Directions](https://mlsanity.com/paper/2107.11508/) (2021)
- [A Perceptual Measure for Evaluating the Resynthesis of Automatic Music Transcriptions](https://mlsanity.com/paper/2202.12257/) (2022)
- [Music Audio-Visual Question Answering Requires Specialized Multimodal Designs](https://mlsanity.com/paper/2505.20638/) (2025)
- [Late multimodal fusion for image and audio music transcription](https://mlsanity.com/paper/2204.03063/) (2022)
- [MusicLIME: Explainable Multimodal Music Understanding](https://mlsanity.com/paper/2409.10496/) (2024)
- [Musical Score Understanding Benchmark: Evaluating Large Language Models' Comprehension of Complete Musical Scores](https://mlsanity.com/paper/2511.20697/) (2025)
- [Multimodal Classification: Current Landscape, Taxonomy and Future Directions](https://mlsanity.com/paper/2109.09020/) (2021)
- [Efficient Cross-Modal Retrieval via Deep Binary Hashing and Quantization](https://mlsanity.com/paper/2202.10232/) (2022)
- [Knowledge-based Multimodal Music Similarity](https://mlsanity.com/paper/2306.12249/) (2023)
- [Music Era Recognition Using Supervised Contrastive Learning and Artist Information](https://mlsanity.com/paper/2407.05368/) (2024)
- [Benchmarking Sub-Genre Classification For Mainstage Dance Music](https://mlsanity.com/paper/2409.06690/) (2024)

## Abstract (arXiv)

Towards improving the performance in various music information processing tasks, recent studies exploit different modalities able to capture diverse aspects of music. Such modalities include audio recordings, symbolic music scores, mid-level representations, motion, and gestural data, video recordings, editorial or cultural tags, lyrics and album cover arts. This paper critically reviews the various approaches adopted in Music Information Processing and Retrieval and highlights how multimodal algorithms can help Music Computing applications. First, we categorize the related literature based on the application they address. Subsequently, we analyze existing information fusion approaches, and we conclude with the set of challenges that Music Information Retrieval and Sound and Music Computing research communities should focus in the next years.