# Time-Varying Audio Effect Modeling by End-to-End Adversarial Training

arXiv: https://arxiv.org/abs/2512.15313 · HTML: https://mlsanity.com/paper/2512.15313/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 35 references that are in the map.
- First posted on arXiv 17 Dec 2025.

## Builds on (most cited first, 17 of 17)

- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [High Fidelity Neural Audio Compression](https://mlsanity.com/paper/2210.13438/) (2022)
- [Towards Principled Methods for Training Generative Adversarial Networks](https://mlsanity.com/paper/1701.04862/) (2017)
- [MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis](https://mlsanity.com/paper/1910.06711/) (2019)
- [FMA: A Dataset For Music Analysis](https://mlsanity.com/paper/1612.01840/) (2016)
- [Mode Seeking Generative Adversarial Networks for Diverse Image Synthesis](https://mlsanity.com/paper/1903.05628/) (2019)
- [Modelling black-box audio effects with time-varying feature modulation](https://mlsanity.com/paper/2211.00497/) (2022)
- [A general-purpose deep learning approach to model time-varying audio effects](https://mlsanity.com/paper/1905.06148/) (2019)
- [Unsupervised Harmonic Parameter Estimation Using Differentiable DSP and Spectral Optimal Transport](https://mlsanity.com/paper/2312.14507/) (2023)
- [Streamable Neural Audio Synthesis With Non-Causal Convolutions](https://mlsanity.com/paper/2204.07064/) (2022)
- [Modulation Extraction for LFO-driven Audio Effects](https://mlsanity.com/paper/2305.13262/) (2023)
- [Adversarial Guitar Amplifier Modelling With Unpaired Data](https://mlsanity.com/paper/2211.00943/) (2022)
- [Differentiable Grey-box Modelling of Phaser Effects using Frame-based Spectral Processing](https://mlsanity.com/paper/2306.01332/) (2023)
- [Improving Unsupervised Clean-to-Rendered Guitar Tone Transformation Using GANs and Integrated Unaligned Clean Data](https://mlsanity.com/paper/2406.15751/) (2024)

## Abstract (arXiv)

Deep learning has become a standard approach for the modeling of audio effects, yet strictly black-box modeling remains problematic for time-varying systems. Unlike time-invariant effects, training models on devices with internal modulation typically requires the recording or extraction of control signals to ensure the time-alignment required by standard loss functions. This paper introduces a Generative Adversarial Network (GAN) framework to model such effects using only input-output audio recordings, without requiring a modulation signal extraction. We propose a convolutional-recurrent architecture trained via a two-stage strategy: an initial adversarial phase allows the model to learn the distribution of the modulation behavior without strict phase constraints, followed by a supervised fine-tuning phase where a State Prediction Network (SPN) estimates the initial internal states required to synchronize the model with the target. Additionally, a new metric based on chirp-train signals is developed to quantify modulation accuracy. Experiments modeling a vintage hardware phaser demonstrate the method's ability to capture time-varying dynamics in a fully black-box context.