# Neural Waveshaping Synthesis

arXiv: https://arxiv.org/abs/2107.05050 · HTML: https://mlsanity.com/paper/2107.05050/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 40 references that are in the map.
- First posted on arXiv 11 Jul 2021.

## Builds on (most cited first, 23 of 23)

- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis](https://mlsanity.com/paper/1910.06711/) (2019)
- [Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram](https://mlsanity.com/paper/1910.11480/) (2019)
- [WaveGlow: A Flow-based Generative Network for Speech Synthesis](https://mlsanity.com/paper/1811.00002/) (2018)
- [Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders](https://mlsanity.com/paper/1704.01279/) (2017)
- [Adversarial Audio Synthesis](https://mlsanity.com/paper/1802.04208/) (2018)
- [Parallel WaveNet: Fast High-Fidelity Speech Synthesis](https://mlsanity.com/paper/1711.10433/) (2017)
- [DDSP: Differentiable Digital Signal Processing](https://mlsanity.com/paper/2001.04643/) (2020)
- [CKConv: Continuous Kernel Convolution For Sequential Data](https://mlsanity.com/paper/2102.02611/) (2021)
- [GANSynth: Adversarial Neural Audio Synthesis](https://mlsanity.com/paper/1902.08710/) (2019)
- [SampleRNN: An Unconditional End-to-End Neural Audio Generation Model](https://mlsanity.com/paper/1612.07837/) (2016)
- [CREPE: A Convolutional Representation for Pitch Estimation](https://mlsanity.com/paper/1802.06182/) (2018)
- [Creating A Multi-track Classical Musical Performance Dataset for Multimodal Music Analysis: Challenges, Insights, and Applications](https://mlsanity.com/paper/1612.08727/) (2016)
- [Neural source-filter waveform models for statistical parametric speech synthesis](https://mlsanity.com/paper/1904.12088/) (2019)
- [DrumGAN: Synthesis of Drum Sounds With Timbral Feature Conditioning Using Generative Adversarial Networks](https://mlsanity.com/paper/2008.12073/) (2020)
- [TimbreTron: A WaveNet(CycleGAN(CQT(Audio))) Pipeline for Musical Timbre Transfer](https://mlsanity.com/paper/1811.09620/) (2018)
- [Neural Music Synthesis for Flexible Timbre Control](https://mlsanity.com/paper/1811.00223/) (2018)
- [Hierarchical Timbre-Painting and Articulation Generation](https://mlsanity.com/paper/2008.13095/) (2020)
- [Transferring neural speech waveform synthesizers to musical instrument sounds generation](https://mlsanity.com/paper/1910.12381/) (2019)

## Built on by (most cited first, 12 of 12)

- [MIDI-DDSP: Detailed Control of Musical Performance via Hierarchical Modeling](https://mlsanity.com/paper/2112.09312/) (2021)
- [Evaluating generative audio systems and their metrics](https://mlsanity.com/paper/2209.00130/) (2022)
- [DDX7: Differentiable FM Synthesis of Musical Instrument Sounds](https://mlsanity.com/paper/2208.06169/) (2022)
- [Deep Performer: Score-to-Audio Music Performance Synthesis](https://mlsanity.com/paper/2202.06034/) (2022)
- [Towards Robust Unsupervised Disentanglement of Sequential Data -- A Case Study Using Music Audio](https://mlsanity.com/paper/2205.05871/) (2022)
- [TokenSynth: A Token-based Neural Synthesizer for Instrument Cloning and Text-to-Instrument](https://mlsanity.com/paper/2502.08939/) (2025)
- [Differentiable Digital Signal Processing Mixture Model for Synthesis Parameter Extraction from Mixture of Harmonic Sounds](https://mlsanity.com/paper/2202.00200/) (2022)
- [Differentiable Modal Synthesis for Physical Modeling of Planar String Sound and Motion Simulation](https://mlsanity.com/paper/2407.05516/) (2024)
- [On the Interplay Between Sparsity, Naturalness, Intelligibility, and Prosody in Speech Synthesis](https://mlsanity.com/paper/2110.01147/) (2021)
- [Real-time Timbre Remapping with Differentiable DSP](https://mlsanity.com/paper/2407.04547/) (2024)
- [Fast, High-Quality and Parameter-Efficient Articulatory Synthesis using Differentiable DSP](https://mlsanity.com/paper/2409.02451/) (2024)
- [Modulation Discovery with Differentiable Digital Signal Processing](https://mlsanity.com/paper/2510.06204/) (2025)

## Abstract (arXiv)

We present the Neural Waveshaping Unit (NEWT): a novel, lightweight, fully causal approach to neural audio synthesis which operates directly in the waveform domain, with an accompanying optimisation (FastNEWT) for efficient CPU inference. The NEWT uses time-distributed multilayer perceptrons with periodic activations to implicitly learn nonlinear transfer functions that encode the characteristics of a target timbre. Once trained, a NEWT can produce complex timbral evolutions by simple affine transformations of its input and output signals. We paired the NEWT with a differentiable noise synthesiser and reverb and found it capable of generating realistic musical instrument performances with only 260k total model parameters, conditioned on F0 and loudness features. We compared our method to state-of-the-art benchmarks with a multi-stimulus listening test and the Fr'echet Audio Distance and found it performed competitively across the tested timbral domains. Our method significantly outperformed the benchmarks in terms of generation speed, and achieved real-time performance on a consumer CPU, both with and without FastNEWT, suggesting it is a viable basis for future creative sound design tools.