# TimbreCLIP: Connecting Timbre to Text and Images

arXiv: https://arxiv.org/abs/2211.11225 · HTML: https://mlsanity.com/paper/2211.11225/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 15 references that are in the map.
- First posted on arXiv 21 Nov 2022.

## Builds on (most cited first, 13 of 13)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [AudioCLIP: Extending CLIP to Image, Text and Audio](https://mlsanity.com/paper/2106.13043/) (2021)
- [VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance](https://mlsanity.com/paper/2204.08583/) (2022)
- [Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders](https://mlsanity.com/paper/1704.01279/) (2017)
- [Wav2CLIP: Learning Robust Audio Representations From CLIP](https://mlsanity.com/paper/2110.11499/) (2021)
- [MuLan: A Joint Embedding of Music Audio and Natural Language](https://mlsanity.com/paper/2208.12415/) (2022)
- [Contrastive Audio-Language Learning for Music](https://mlsanity.com/paper/2208.12208/) (2022)
- [SpeechCLIP: Integrating Speech with Pre-Trained Vision and Language Model](https://mlsanity.com/paper/2210.00705/) (2022)
- [One Billion Audio Sounds from GPU-enabled Modular Synthesis](https://mlsanity.com/paper/2104.12922/) (2021)
- [Robust Sound-Guided Image Manipulation](https://mlsanity.com/paper/2208.14114/) (2022)

## Built on by (most cited first, 5 of 5)

- [Soundini: Sound-Guided Diffusion for Natural Video Editing](https://mlsanity.com/paper/2304.06818/) (2023)
- [SynthScribe: Deep Multimodal Tools for Synthesizer Sound Retrieval and Exploration](https://mlsanity.com/paper/2312.04690/) (2023)
- [Contrastive Learning from Synthetic Audio Doppelg"angers](https://mlsanity.com/paper/2406.05923/) (2024)
- [Audio Latent Space Cartography](https://mlsanity.com/paper/2212.02610/) (2022)
- [Testing chatbots on the creation of encoders for audio conditioned image generation](https://mlsanity.com/paper/2509.09717/) (2025)

## Abstract (arXiv)

We present work in progress on TimbreCLIP, an audio-text cross modal embedding trained on single instrument notes. We evaluate the models with a cross-modal retrieval task on synth patches. Finally, we demonstrate the application of TimbreCLIP on two tasks: text-driven audio equalization and timbre to image generation.