# Sketching With Your Voice: "Non-Phonorealistic" Rendering of Sounds via Vocal Imitation

arXiv: https://arxiv.org/abs/2409.13507 · HTML: https://mlsanity.com/paper/2409.13507/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 96 references that are in the map.
- First posted on arXiv 20 Sep 2024.

## Builds on (most cited first, 20 of 20)

- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [DiffWave: A Versatile Diffusion Model for Audio Synthesis](https://mlsanity.com/paper/2009.09761/) (2020)
- [Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale](https://mlsanity.com/paper/2306.15687/) (2023)
- [FSD50K: An Open Dataset of Human-Labeled Sound Events](https://mlsanity.com/paper/2010.00475/) (2020)
- [Visually Indicated Sounds](https://mlsanity.com/paper/1512.08512/) (2015)
- [SketchyGAN: Towards Diverse and Realistic Sketch to Image Synthesis](https://mlsanity.com/paper/1801.02753/) (2018)
- [Scribbler: Controlling Deep Image Synthesis with Sketch and Color](https://mlsanity.com/paper/1612.00835/) (2016)
- [Reasoning About Pragmatics with Neural Listeners and Speakers](https://mlsanity.com/paper/1604.00562/) (2016)
- [CLIPasso: Semantically-Aware Object Sketching](https://mlsanity.com/paper/2202.05822/) (2022)
- [Sketch-based 3D Shape Retrieval using Convolutional Neural Networks](https://mlsanity.com/paper/1504.03504/) (2015)
- [Program Synthesis with Pragmatic Communication](https://mlsanity.com/paper/2007.05060/) (2020)
- [Virtuoso: Massive Multilingual Speech-Text Joint Semi-Supervised Learning for Text-To-Speech](https://mlsanity.com/paper/2210.15447/) (2022)
- [SEVA: Leveraging sketches to evaluate alignment between human and machine visual abstraction](https://mlsanity.com/paper/2312.03035/) (2023)
- [Designing Perceptual Puzzles by Differentiating Probabilistic Programs](https://mlsanity.com/paper/2204.12301/) (2022)
- [Block and Detail: Scaffolding Sketch-to-Image Generation](https://mlsanity.com/paper/2402.18116/) (2024)
- [Line Drawings from 3D Models](https://mlsanity.com/paper/1810.01175/) (2018)
- [Acting as Inverse Inverse Planning](https://mlsanity.com/paper/2305.16913/) (2023)
- [Onoma-to-wave: Environmental sound synthesis from onomatopoeic words](https://mlsanity.com/paper/2102.05872/) (2021)
- [Learning Evolved Combinatorial Symbols with a Neuro-symbolic Generative Model](https://mlsanity.com/paper/2104.08274/) (2021)
- [Visual onoma-to-wave: environmental sound synthesis from visual onomatopoeias and sound-source images](https://mlsanity.com/paper/2210.09173/) (2022)

## Built on by (most cited first, 1 of 1)

- [Reconstructing the Vocal Tract with Differentiable Acoustic Simulation](https://mlsanity.com/paper/2609.36737/) (2026)

## Abstract (arXiv)

We present a method for automatically producing human-like vocal imitations of sounds: the equivalent of "sketching," but for auditory rather than visual representation. Starting with a simulated model of the human vocal tract, we first try generating vocal imitations by tuning the model's control parameters to make the synthesized vocalization match the target sound in terms of perceptually-salient auditory features. Then, to better match human intuitions, we apply a cognitive theory of communication to take into account how human speakers reason strategically about their listeners. Finally, we show through several experiments and user studies that when we add this type of communicative reasoning to our method, it aligns with human intuitions better than matching auditory features alone does. This observation has broad implications for the study of depiction in computer graphics.