# SynthScribe: Deep Multimodal Tools for Synthesizer Sound Retrieval and Exploration

arXiv: https://arxiv.org/abs/2312.04690 · HTML: https://mlsanity.com/paper/2312.04690/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 40 references that are in the map.
- First posted on arXiv 7 Dec 2023.

## Builds on (most cited first, 10 of 10)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [TimbreTron: A WaveNet(CycleGAN(CQT(Audio))) Pipeline for Musical Timbre Transfer](https://mlsanity.com/paper/1811.09620/) (2018)
- [Universal audio synthesizer control with normalizing flows](https://mlsanity.com/paper/1907.00971/) (2019)
- [An intuitive control space for material appearance](https://mlsanity.com/paper/1806.04950/) (2018)
- [DawDreamer: Bridging the Gap Between Digital Audio Workstations and Python Interfaces](https://mlsanity.com/paper/2111.09931/) (2021)
- [TimbreCLIP: Connecting Timbre to Text and Images](https://mlsanity.com/paper/2211.11225/) (2022)

## Built on by (most cited first, 3 of 3)

- [Neural Proxies for Sound Synthesizers: Learning Perceptually Informed Preset Representations](https://mlsanity.com/paper/2509.07635/) (2025)
- [TimberAgent: Gram-Guided Retrieval for Executable Music Effect Control](https://mlsanity.com/paper/2603.09332/) (2026)
- [A Text-Steerable Instrument for Sketching Procedural Soundscapes via Language Models](https://mlsanity.com/paper/2607.00309/) (2026)

## Abstract (arXiv)

Synthesizers are powerful tools that allow musicians to create dynamic and original sounds. Existing commercial interfaces for synthesizers typically require musicians to interact with complex low-level parameters or to manage large libraries of premade sounds. To address these challenges, we implement SynthScribe -- a fullstack system that uses multimodal deep learning to let users express their intentions at a much higher level. We implement features which address a number of difficulties, namely 1) searching through existing sounds, 2) creating completely new sounds, 3) making meaningful modifications to a given sound. This is achieved with three main features: a multimodal search engine for a large library of synthesizer sounds; a user centered genetic algorithm by which completely new sounds can be created and selected given the users preferences; a sound editing support feature which highlights and gives examples for key control parameters with respect to a text or audio based query. The results of our user studies show SynthScribe is capable of reliably retrieving and modifying sounds while also affording the ability to create completely new sounds that expand a musicians creative horizon.