# Zero-Effort Image-to-Music Generation: An Interpretable RAG-based VLM Approach

arXiv: https://arxiv.org/abs/2509.22378 · HTML: https://mlsanity.com/paper/2509.22378/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 39 references that are in the map.
- First posted on arXiv 26 Sep 2025.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge](https://mlsanity.com/paper/2411.16594/) (2024)
- [Long-CLIP: Unlocking the Long-Text Capability of CLIP](https://mlsanity.com/paper/2403.15378/) (2024)
- [Mustango: Toward Controllable Text-to-Music Generation](https://mlsanity.com/paper/2311.08355/) (2023)
- [Building a Large Scale Dataset for Image Emotion Recognition: The Fine Print and The Benchmark](https://mlsanity.com/paper/1605.02677/) (2016)
- [Video Background Music Generation with Controllable Music Transformer](https://mlsanity.com/paper/2111.08380/) (2021)
- [Kwai Keye-VL Technical Report](https://mlsanity.com/paper/2507.01949/) (2025)
- [MusPy: A Toolkit for Symbolic Music Generation](https://mlsanity.com/paper/2008.01951/) (2020)
- [Video Background Music Generation: Dataset, Method and Evaluation](https://mlsanity.com/paper/2211.11248/) (2022)
- [SongEval: A Benchmark Dataset for Song Aesthetics Evaluation](https://mlsanity.com/paper/2505.10793/) (2025)
- [MidiCaps: A large-scale MIDI dataset with text captions](https://mlsanity.com/paper/2406.02255/) (2024)
- [MeLFusion: Synthesizing Music from Image and Language Cues using Diffusion Models](https://mlsanity.com/paper/2406.04673/) (2024)
- [An overview of domain-specific foundation model: key technologies, applications and challenges](https://mlsanity.com/paper/2409.04267/) (2024)
- [XMusic: Towards a Generalized and Controllable Symbolic Music Generation Framework](https://mlsanity.com/paper/2501.08809/) (2025)
- [PianoBART: Symbolic Piano Music Generation and Understanding with Large-Scale Pre-Training](https://mlsanity.com/paper/2407.03361/) (2024)
- [Towards Advanced Mathematical Reasoning for LLMs via First-Order Logic Theorem Proving](https://mlsanity.com/paper/2506.17104/) (2025)
- [Mozart's Touch: A Lightweight Multi-modal Music Generation Framework Based on Pre-Trained Large Models](https://mlsanity.com/paper/2405.02801/) (2024)
- [Bridging Paintings and Music -- Exploring Emotion based Music Generation through Paintings](https://mlsanity.com/paper/2409.07827/) (2024)
- [Multimodal Music Generation with Explicit Bridges and Retrieval Augmentation](https://mlsanity.com/paper/2412.09428/) (2024)
- [Vision-to-Music Generation: A Survey](https://mlsanity.com/paper/2503.21254/) (2025)
- [A Survey on Music Generation from Single-Modal, Cross-Modal, and Multi-Modal Perspectives](https://mlsanity.com/paper/2504.00837/) (2025)
- [Automatic Stage Lighting Control: Is it a Rule-Driven Process or Generative Task?](https://mlsanity.com/paper/2506.01482/) (2025)
- [Emotion-Guided Image to Music Generation](https://mlsanity.com/paper/2410.22299/) (2024)

## Abstract (arXiv)

Recently, Image-to-Music (I2M) generation has garnered significant attention, with potential applications in fields such as gaming, advertising, and multi-modal art creation. However, due to the ambiguous and subjective nature of I2M tasks, most end-to-end methods lack interpretability, leaving users puzzled about the generation results. Even methods based on emotion mapping face controversy, as emotion represents only a singular aspect of art. Additionally, most learning-based methods require substantial computational resources and large datasets for training, hindering accessibility for common users. To address these challenges, we propose the first Vision Language Model (VLM)-based I2M framework that offers high interpretability and low computational cost. Specifically, we utilize ABC notation to bridge the text and music modalities, enabling the VLM to generate music using natural language. We then apply multi-modal Retrieval-Augmented Generation (RAG) and self-refinement techniques to allow the VLM to produce high-quality music without external training. Furthermore, we leverage the generated motivations in text and the attention maps from the VLM to provide explanations for the generated results in both text and image modalities. To validate our method, we conduct both human studies and machine evaluations, where our method outperforms others in terms of music quality and music-image consistency, indicating promising results. Our code is available at https://github.com/RS2002/Image2Music .