# Transferring neural speech waveform synthesizers to musical instrument sounds generation

arXiv: https://arxiv.org/abs/1910.12381 · HTML: https://mlsanity.com/paper/1910.12381/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Oct 2019.

## Built on by (most cited first, 4 of 4)

- [Neural Waveshaping Synthesis](https://mlsanity.com/paper/2107.05050/) (2021)
- [CycleDRUMS: Automatic Drum Arrangement For Bass Lines Using CycleGAN](https://mlsanity.com/paper/2104.00353/) (2021)
- [Fast, High-Quality and Parameter-Efficient Articulatory Synthesis using Differentiable DSP](https://mlsanity.com/paper/2409.02451/) (2024)
- [EVA-GAN: Enhanced Various Audio Generation via Scalable Generative Adversarial Networks](https://mlsanity.com/paper/2402.00892/) (2024)

## Abstract (arXiv)

Recent neural waveform synthesizers such as WaveNet, WaveGlow, and the neural-source-filter (NSF) model have shown good performance in speech synthesis despite their different methods of waveform generation. The similarity between speech and music audio synthesis techniques suggests interesting avenues to explore in terms of the best way to apply speech synthesizers in the music domain. This work compares three neural synthesizers used for musical instrument sounds generation under three scenarios: training from scratch on music data, zero-shot learning from the speech domain, and fine-tuning-based adaptation from the speech to the music domain. The results of a large-scale perceptual test demonstrated that the performance of three synthesizers improved when they were pre-trained on speech data and fine-tuned on music data, which indicates the usefulness of knowledge from speech data for music audio generation. Among the synthesizers, WaveGlow showed the best potential in zero-shot learning while NSF performed best in the other scenarios and could generate samples that were perceptually close to natural audio.