# MusiXQA: Advancing Visual Music Understanding in Multimodal Large Language Models

arXiv: https://arxiv.org/abs/2506.23009 · HTML: https://mlsanity.com/paper/2506.23009/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Audio-language model reasoning, one of 17 directions in LLM reasoning.
- Builds on 34 of its 51 references that are in the map.
- First posted on arXiv 28 Jun 2025.

## Placed in

- LLM reasoning / [Audio-language model reasoning](https://mlsanity.com/lr/audio-language-model-reasoning/)
- Multimodal LLMs / vision-language models / [Video understanding with MLLMs](https://mlsanity.com/mllm/video-understanding-with-mllms/)

## Builds on (most cited first, 30 of 34)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI](https://mlsanity.com/paper/2311.16502/) (2023)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models](https://mlsanity.com/paper/2305.07895/) (2023)
- [MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark](https://mlsanity.com/paper/2402.04788/) (2024)
- [LayoutGPT: Compositional Visual Planning and Generation with Large Language Models](https://mlsanity.com/paper/2305.15393/) (2023)
- [mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understanding](https://mlsanity.com/paper/2403.12895/) (2024)
- [PaliGemma 2: A Family of Versatile VLMs for Transfer](https://mlsanity.com/paper/2412.03555/) (2024)
- [TextOCR: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text](https://mlsanity.com/paper/2105.05486/) (2021)
- [PP-OCR: A Practical Ultra Lightweight OCR System](https://mlsanity.com/paper/2009.09941/) (2020)
- [GUI Agents: A Survey](https://mlsanity.com/paper/2412.13501/) (2024)
- [Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey](https://mlsanity.com/paper/2411.17558/) (2024)
- [MLLM-Tool: A Multimodal Large Language Model For Tool Agent Learning](https://mlsanity.com/paper/2401.10727/) (2024)
- [ChatMusician: Understanding and Generating Music Intrinsically with LLM](https://mlsanity.com/paper/2402.16153/) (2024)
- [Understanding Optical Music Recognition](https://mlsanity.com/paper/1908.03608/) (2019)
- [MusicAgent: An AI Agent for Music Understanding and Generation with Large Language Models](https://mlsanity.com/paper/2310.11954/) (2023)
- [Sheet Music Transformer: End-To-End Optical Music Recognition Beyond Monophonic Transcription](https://mlsanity.com/paper/2402.07596/) (2024)
- [DeepScores -- A Dataset for Segmentation, Detection and Classification of Tiny Objects](https://mlsanity.com/paper/1804.00525/) (2018)
- [End-to-End Full-Page Optical Music Recognition for Pianoform Sheet Music](https://mlsanity.com/paper/2405.12105/) (2024)
- [Optical Music Recognition with Convolutional Sequence-to-Sequence Models](https://mlsanity.com/paper/1707.04877/) (2017)
- [Optical Music Recognition: State of the Art and Major Challenges](https://mlsanity.com/paper/2006.07885/) (2020)
- [LLaVA-Read: Enhancing Reading Ability of Multimodal Language Models](https://mlsanity.com/paper/2407.19185/) (2024)
- [DoReMi: First glance at a universal OMR dataset](https://mlsanity.com/paper/2107.07786/) (2021)
- [Practical End-to-End Optical Music Recognition for Pianoform Music](https://mlsanity.com/paper/2403.13763/) (2024)
- [TRINS: Towards Multimodal Language Models that Can Read](https://mlsanity.com/paper/2406.06730/) (2024)
- [TextLap: Customizing Language Models for Text-to-Layout Planning](https://mlsanity.com/paper/2410.12844/) (2024)
- [A Unified Representation Framework for the Evaluation of Optical Music Recognition Systems](https://mlsanity.com/paper/2312.12908/) (2023)

## Built on by (most cited first, 4 of 4)

- [Towards an AI Musician: Synthesizing Sheet Music Problems for Musical Reasoning](https://mlsanity.com/paper/2509.04059/) (2025)
- [From Image Generation to Infrastructure Design: a Multi-agent Pipeline for Street Design Generation](https://mlsanity.com/paper/2509.05469/) (2025)
- [Musical Score Understanding Benchmark: Evaluating Large Language Models' Comprehension of Complete Musical Scores](https://mlsanity.com/paper/2511.20697/) (2025)
- [Direct content-based retrieval from music scores images](https://mlsanity.com/paper/2605.22255/) (2026)

## Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have achieved remarkable visual reasoning abilities in natural images, text-rich documents, and graphic designs. However, their ability to interpret music sheets remains underexplored. To bridge this gap, we introduce MusiXQA, the first comprehensive dataset for evaluating and advancing MLLMs in music sheet understanding. MusiXQA features high-quality synthetic music sheets generated via MusiXTeX, with structured annotations covering note pitch and duration, chords, clefs, key/time signatures, and text, enabling diverse visual QA tasks. Through extensive evaluations, we reveal significant limitations of current state-of-the-art MLLMs in this domain. Beyond benchmarking, we developed Phi-3-MusiX, an MLLM fine-tuned on our dataset, achieving significant performance gains over GPT-based methods. The proposed dataset and model establish a foundation for future advances in MLLMs for music sheet understanding. Code, data, and model will be released upon acceptance.