mlsanityData to 14 Oct 2026

A Multimodal Symphony: Integrating Taste and Sound through Generative AI

arXiv 2503.02823 ↗First posted 4 Mar 2025cs.SD cs.AI cs.MMcited by 4

Not on the map. Why?

Abstract (arXiv)

In recent decades, neuroscientific and psychological research has traced direct relationships between taste and auditory perceptions. This article explores multimodal generative models capable of converting taste information into music, building on this foundational research. We provide a brief review of the state of the art in this field, highlighting key findings and methodologies. We present an experiment in which a fine-tuned version of a generative music model (MusicGEN) is used to generate music based on detailed taste descriptions provided for each musical piece. The results are promising: according the participants' (n=111) evaluation, the fine-tuned model produces music that more coherently reflects the input taste descriptions compared to the non-fine-tuned model. This study represents a significant step towards understanding and developing embodied interactions between AI, sound, and taste, opening new possibilities in the field of generative AI. We release our dataset, code and pre-trained model at: https://osf.io/xs5jy/.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 18 of its 74 references that are in the map. First posted on arXiv 4 Mar 2025.

Builds on

18 of 74 references.

References · 18

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  6. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  7. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  8. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  9. 2022High Fidelity Neural Audio Compressioncited by 687
  10. 2023Simple and Controllable Music Generationcited by 488
  11. 2023MusicLM: Generating Music From Textcited by 448
  12. 2023Adapting Frechet Audio Distance for Generative Music Evaluationcited by 92
6 more
  1. 2023Make-An-Audio 2: Temporal-Enhanced Text-to-Audio Generationcited by 90
  2. 2023Let the LLMs Talk: Simulating Human-to-Human Conversational QA via Zero-Shot LLM-to-LLM Interactionscited by 70
  3. 2019A Comparative Study on End-to-end Speech to Text Translationcited by 40
  4. 2019Affective Computing for Large-Scale Heterogeneous Multimedia Data: A Surveycited by 18
  5. 2024Evaluating Human Alignment and Model Faithfulness of LLM Rationalecited by 12
  6. 2020Emotion-Based End-to-End Matching Between Image and Music in Valence-Arousal Spacecited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Multimodal Dataset Normalization and Perceptual Validation for Music-Taste Correspondencescited by 2
  2. 2026Taste-aware music retrieval from audio embeddingscited by 1
  3. 2026Can LLMs understand LilyPond? A benchmark for symbolic music generation and understandingcited by 0
  4. 2026A Quantized Native Runtime for On-Device Semantic Audio Generationcited by 0