# Hierarchical Timbre-Painting and Articulation Generation

arXiv: https://arxiv.org/abs/2008.13095 · HTML: https://mlsanity.com/paper/2008.13095/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Aug 2020.

## Built on by (most cited first, 9 of 9)

- [Catch-A-Waveform: Learning to Generate Audio from a Single Short Example](https://mlsanity.com/paper/2106.06426/) (2021)
- [DDX7: Differentiable FM Synthesis of Musical Instrument Sounds](https://mlsanity.com/paper/2208.06169/) (2022)
- [Neural Waveshaping Synthesis](https://mlsanity.com/paper/2107.05050/) (2021)
- [Timbre Transfer with Variational Auto Encoding and Cycle-Consistent Adversarial Networks](https://mlsanity.com/paper/2109.02096/) (2021)
- [Timbre transfer using image-to-image denoising diffusion implicit models](https://mlsanity.com/paper/2307.04586/) (2023)
- [Differentiable Digital Signal Processing Mixture Model for Synthesis Parameter Extraction from Mixture of Harmonic Sounds](https://mlsanity.com/paper/2202.00200/) (2022)
- [Fast, High-Quality and Parameter-Efficient Articulatory Synthesis using Differentiable DSP](https://mlsanity.com/paper/2409.02451/) (2024)
- [A Survey of Music Generation in the Context of Interaction](https://mlsanity.com/paper/2402.15294/) (2024)
- [WaveTransfer: A Flexible End-to-end Multi-instrument Timbre Transfer with Diffusion](https://mlsanity.com/paper/2409.15321/) (2024)

## Abstract (arXiv)

We present a fast and high-fidelity method for music generation, based on specified f0 and loudness, such that the synthesized audio mimics the timbre and articulation of a target instrument. The generation process consists of learned source-filtering networks, which reconstruct the signal at increasing resolutions. The model optimizes a multi-resolution spectral loss as the reconstruction loss, an adversarial loss to make the audio sound more realistic, and a perceptual f0 loss to align the output to the desired input pitch contour. The proposed architecture enables high-quality fitting of an instrument, given a sample that can be as short as a few minutes, and the method demonstrates state-of-the-art timbre transfer capabilities. Code and audio samples are shared at https://github.com/mosheman5/timbre_painting.