# Sketch2Sound: Controllable Audio Generation via Time-Varying Signals and Sonic Imitations

arXiv: https://arxiv.org/abs/2412.08550 · HTML: https://mlsanity.com/paper/2412.08550/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Dec 2024.

## Built on by (most cited first, 24 of 24)

- [Coherent Audio-Visual Editing via Conditional Audio Generation Following Video Edits](https://mlsanity.com/paper/2512.07209/) (2025)
- [Stable Audio 3](https://mlsanity.com/paper/2605.17991/) (2026)
- [ReelWave: Multi-Agentic Movie Sound Generation through Multimodal LLM Conversation](https://mlsanity.com/paper/2503.07217/) (2025)
- [LiLAC: A Lightweight Latent ControlNet for Musical Audio Generation](https://mlsanity.com/paper/2506.11476/) (2025)
- [Post-Training Quantization for Audio Diffusion Transformers](https://mlsanity.com/paper/2510.00313/) (2025)
- [Hear What Matters! Text-conditioned Selective Video-to-Audio Generation](https://mlsanity.com/paper/2512.02650/) (2025)
- [Step-by-Step Video-to-Audio Synthesis via Negative Audio Guidance](https://mlsanity.com/paper/2506.20995/) (2025)
- [User-guided Generative Source Separation](https://mlsanity.com/paper/2507.01339/) (2025)
- [Recomposer: Event-roll-guided generative audio editing](https://mlsanity.com/paper/2509.05256/) (2025)
- [Low-Resource Guidance for Controllable Latent Audio Diffusion](https://mlsanity.com/paper/2603.04366/) (2026)
- [SynSonic: Augmenting Sound Event Detection through Text-to-Audio Diffusion ControlNet and Effective Sample Filtering](https://mlsanity.com/paper/2509.18603/) (2025)
- [Audio Palette: A Diffusion Transformer with Multi-Signal Conditioning for Controllable Foley Synthesis](https://mlsanity.com/paper/2510.12175/) (2025)
- [Live Music Diffusion Models: Efficient Fine-Tuning and Post-Training of Interactive Diffusion Music Generators](https://mlsanity.com/paper/2605.22717/) (2026)
- [EXPOTION: Facial Expression and Motion Control for Multimodal Music Generation](https://mlsanity.com/paper/2507.04955/) (2025)
- [Single-step Controllable Music Bandwidth Extension With Flow Matching](https://mlsanity.com/paper/2601.14356/) (2026)
- [Mix2Morph: Learning Sound Morphing from Noisy Mixes](https://mlsanity.com/paper/2601.20426/) (2026)
- [Latent Fourier Transform](https://mlsanity.com/paper/2604.17986/) (2026)
- [Break-the-Beat! Controllable MIDI-to-Drum Audio Synthesis](https://mlsanity.com/paper/2605.14555/) (2026)
- [Taming Audio VAEs via Target-KL Regularization](https://mlsanity.com/paper/2605.17085/) (2026)
- [FoleyGenEx: Unified Video-to-Audio Generation with Multi-Modal Control, Temporal Alignment, and Semantic Precision](https://mlsanity.com/paper/2606.14049/) (2026)
- [A Production-Oriented Framework for Evaluation of SFX Generation](https://mlsanity.com/paper/2607.09973/) (2026)
- [Architecture and Affordances of PLAUD: Performative Latents and Unsupervised DDSP](https://mlsanity.com/paper/2608.13724/) (2026)
- [SCAPES: Semantically Conditioned Autoregressive Prior for Environmental Sounds](https://mlsanity.com/paper/2609.04634/) (2026)
- [PhysicsSliders: Learning Physical Controls for Audio Foley Generation](https://mlsanity.com/paper/2610.18597/) (2026)

## Abstract (arXiv)

We present Sketch2Sound, a generative audio model capable of creating high-quality sounds from a set of interpretable time-varying control signals: loudness, brightness, and pitch, as well as text prompts. Sketch2Sound can synthesize arbitrary sounds from sonic imitations (i.e.,~a vocal imitation or a reference sound-shape). Sketch2Sound can be implemented on top of any text-to-audio latent diffusion transformer (DiT), and requires only 40k steps of fine-tuning and a single linear layer per control, making it more lightweight than existing methods like ControlNet. To synthesize from sketchlike sonic imitations, we propose applying random median filters to the control signals during training, allowing Sketch2Sound to be prompted using controls with flexible levels of temporal specificity. We show that Sketch2Sound can synthesize sounds that follow the gist of input controls from a vocal imitation while retaining the adherence to an input text prompt and audio quality compared to a text-only baseline. Sketch2Sound allows sound artists to create sounds with the semantic flexibility of text prompts and the expressivity and precision of a sonic gesture or vocal imitation. Sound examples are available at https://hugofloresgarcia.art/sketch2sound/.