mlsanityData to 7 Oct 2026

Bridging Paintings and Music -- Exploring Emotion based Music Generation through Paintings

arXiv 2409.07827 ↗First posted 12 Sep 2024cs.SD cs.CV cs.MMcited by 6

Not on the map. Why?

Abstract (arXiv)

Rapid advancements in artificial intelligence have significantly enhanced generative tasks involving music and images, employing both unimodal and multimodal approaches. This research develops a model capable of generating music that resonates with the emotions depicted in visual arts, integrating emotion labeling, image captioning, and language models to transform visual inputs into musical compositions. Addressing the scarcity of aligned art and music data, we curated the Emotion Painting Music Dataset, pairing paintings with corresponding music for effective training and evaluation. Our dual-stage framework converts images to text descriptions of emotional content and then transforms these descriptions into music, facilitating efficient learning with minimal data. Performance is evaluated using metrics such as Fr'echet Audio Distance (FAD), Total Harmonic Distortion (THD), Inception Score (IS), and KL divergence, with audio-emotion text similarity confirmed by the pre-trained CLAP model to demonstrate high alignment between generated music and text. This synthesis tool bridges visual art and music, enhancing accessibility for the visually impaired and opening avenues in educational and therapeutic applications by providing enriched multi-sensory experiences.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 29 of its 44 references that are in the map. First posted on arXiv 12 Sep 2024.

Builds on

29 of 44 references.

References · 29

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  6. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  7. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Modelscited by 592
  8. 2023MusicLM: Generating Music From Textcited by 442
  9. 2016SeqGAN: Sequence Generative Adversarial Nets with Policy Gradientcited by 375
  10. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretrainingcited by 343
  11. 2018Fr'echet Audio Distance: A Metric for Evaluating Music Enhancement Algorithmscited by 221
  12. 2017MuseGAN: Multi-track Sequential Generative Adversarial Networks for Symbolic Music Generation and Accompanimentcited by 151
17 more
  1. 2018A Hierarchical Latent Vector Model for Learning Long-Term Structure in Musiccited by 131
  2. 2023MusicLDM: Enhancing Novelty in Text-to-Music Generation Using Beat-Synchronous Mixup Strategiescited by 94
  3. 2022I Hear Your True Colors: Image Guided Audio Generationcited by 92
  4. 2023Mousai: Text-to-Music Generation with Long-Context Latent Diffusioncited by 77
  5. 2017MidiNet: A Convolutional Generative Adversarial Network for Symbolic-domain Music Generationcited by 72
  6. 2023Efficient Neural Music Generationcited by 50
  7. 2023WavJourney: Compositional Audio Creation with Large Language Modelscited by 27
  8. 2024MeLFusion: Synthesizing Music from Image and Language Cues using Diffusion Modelscited by 19
  9. 2020From Artificial Neural Networks to Deep Learning for Music Generation -- History, Concepts and Trendscited by 14
  10. 2023Symbolic Music Representations for Classification Tasks: A Systematic Evaluationcited by 14
  11. 2024WavCraft: Audio Editing and Generation with Large Language Modelscited by 14
  12. 2024DExter: Learning and Controlling Performance Expression with Diffusion Modelscited by 11
  13. 2023Leveraging Pre-trained AudioLDM for Sound Generation: A Benchmark Studycited by 10
  14. 2023Adapting Language-Audio Models as Few-Shot Audio Learnerscited by 10
  15. 2023Acoustic Prompt Tuning: Empowering Large Language Models with Audition Capabilitiescited by 10
  16. 2024Mind the Domain Gap: a Systematic Analysis on Bioacoustic Sound Event Detectioncited by 7
  17. 2022cMelGAN: An Efficient Conditional Generative Model Based on Mel Spectrogramscited by 2

Built on by

Cited by year

Citing papers per year: 2025: 6.

6 papers.

  1. 2025MusFlow: Multimodal Music Generation via Conditional Flow Matchingcited by 4
  2. 2025MusicAIR: A Multimodal AI Music Generation Framework Powered by an Algorithm-Driven Corecited by 1
  3. 2025Emotion-Aware Speech Generation with Character-Specific Voices for Comicscited by 0
  4. 2025EmoHeal: An End-to-End System for Personalized Therapeutic Music Retrieval from Fine-grained Emotionscited by 0
  5. 2025Zero-Effort Image-to-Music Generation: An Interpretable RAG-based VLM Approachcited by 0
  6. 2025Art2Music: Generating Music for Art Images with Multi-modal Feeling Alignmentcited by 0