# Streamable Neural Audio Synthesis With Non-Causal Convolutions

arXiv: https://arxiv.org/abs/2204.07064 · HTML: https://mlsanity.com/paper/2204.07064/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 15 references that are in the map.
- First posted on arXiv 14 Apr 2022.

## Builds on (most cited first, 12 of 12)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [SoundStream: An End-to-End Neural Audio Codec](https://mlsanity.com/paper/2107.03312/) (2021)
- [Jukebox: A Generative Model for Music](https://mlsanity.com/paper/2005.00341/) (2020)
- [MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis](https://mlsanity.com/paper/1910.06711/) (2019)
- [DDSP: Differentiable Digital Signal Processing](https://mlsanity.com/paper/2001.04643/) (2020)
- [Wave-U-Net: A Multi-Scale Neural Network for End-to-End Audio Source Separation](https://mlsanity.com/paper/1806.03185/) (2018)
- [RAVE: A variational autoencoder for fast and high-quality neural audio synthesis](https://mlsanity.com/paper/2111.05011/) (2021)
- [ClariNet: Parallel Wave Generation in End-to-End Text-to-Speech](https://mlsanity.com/paper/1807.07281/) (2018)
- [Streaming keyword spotting on mobile devices](https://mlsanity.com/paper/2005.06720/) (2020)
- [TimbreTron: A WaveNet(CycleGAN(CQT(Audio))) Pipeline for Musical Timbre Transfer](https://mlsanity.com/paper/1811.09620/) (2018)
- [SING: Symbol-to-Instrument Neural Generator](https://mlsanity.com/paper/1810.09785/) (2018)

## Built on by (most cited first, 7 of 7)

- [Multi-instrument Music Synthesis with Spectrogram Diffusion](https://mlsanity.com/paper/2206.05408/) (2022)
- [Reimagining Speech: A Scoping Review of Deep Learning-Powered Voice Conversion](https://mlsanity.com/paper/2311.08104/) (2023)
- [PESTO: Real-Time Pitch Estimation with Self-supervised Transposition-equivariant Objective](https://mlsanity.com/paper/2508.01488/) (2025)
- [Designing Neural Synthesizers for Low-Latency Interaction](https://mlsanity.com/paper/2503.11562/) (2025)
- [Empirical Results for Adjusting Truncated Backpropagation Through Time while Training Neural Audio Effects](https://mlsanity.com/paper/2512.07393/) (2025)
- [Time-Varying Audio Effect Modeling by End-to-End Adversarial Training](https://mlsanity.com/paper/2512.15313/) (2025)
- [Feature Encoding in VAE-based Audio Decoders: Effects of Input, Depth and Distribution](https://mlsanity.com/paper/2610.07966/) (2026)

## Abstract (arXiv)

Deep learning models are mostly used in an offline inference fashion. However, this strongly limits the use of these models inside audio generation setups, as most creative workflows are based on real-time digital signal processing. Although approaches based on recurrent networks can be naturally adapted to this buffer-based computation, the use of convolutions still poses some serious challenges. To tackle this issue, the use of causal streaming convolutions have been proposed. However, this requires specific complexified training and can impact the resulting audio quality. In this paper, we introduce a new method allowing to produce non-causal streaming models. This allows to make any convolutional model compatible with real-time buffer-based processing. As our method is based on a post-training reconfiguration of the model, we show that it is able to transform models trained without causal constraints into a streaming model. We show how our method can be adapted to fit complex architectures with parallel branches. To evaluate our method, we apply it on the recent RAVE model, which provides high-quality real-time audio synthesis. We test our approach on multiple music and speech datasets and show that it is faster than overlap-add methods, while having no impact on the generation quality. Finally, we introduce two open-source implementation of our work as Max/MSP and PureData externals, and as a VST audio plugin. This allows to endow traditional digital audio workstation with real-time neural audio synthesis on a laptop CPU.