# The artificial synesthete: Image-melody translations with variational autoencoders

arXiv: https://arxiv.org/abs/2112.02953 · HTML: https://mlsanity.com/paper/2112.02953/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 48 references that are in the map.
- First posted on arXiv 6 Dec 2021.

## Builds on (most cited first, 9 of 9)

- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Autoencoding beyond pixels using a learned similarity metric](https://mlsanity.com/paper/1512.09300/) (2015)
- [A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music](https://mlsanity.com/paper/1803.05428/) (2018)
- [Building a Large Scale Dataset for Image Emotion Recognition: The Fine Print and The Benchmark](https://mlsanity.com/paper/1605.02677/) (2016)
- [Deep Learning Techniques for Music Generation -- A Survey](https://mlsanity.com/paper/1709.01620/) (2017)
- [AI Methods in Algorithmic Composition: A Comprehensive Survey](https://mlsanity.com/paper/1402.0585/) (2014)
- [Translating Visual Art into Music](https://mlsanity.com/paper/1909.01218/) (2019)
- [Autoencoders for music sound modeling: a comparison of linear, shallow, deep, recurrent and variational models](https://mlsanity.com/paper/1806.04096/) (2018)

## Abstract (arXiv)

Abstract This project presents a system of neural networks to translate between images and melodies. Autoencoders compress the information in samples to abstract representation. A translation network learns a set of correspondences between musical and visual concepts from repeated joint exposure. The resulting "artificial synesthete" generates simple melodies inspired by images, and images from music. These are novel interpretation (not transposed data), expressing the machine' perception and understanding. Observing the work, one explores the machine's perception and thus, by contrast, one's own.