# GANStrument: Adversarial Instrument Sound Synthesis with Pitch-invariant Instance Conditioning

arXiv: https://arxiv.org/abs/2211.05385 · HTML: https://mlsanity.com/paper/2211.05385/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 26 references that are in the map.
- First posted on arXiv 10 Nov 2022.

## Builds on (most cited first, 21 of 21)

- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [GAN Inversion: A Survey](https://mlsanity.com/paper/2101.05278/) (2021)
- [MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis](https://mlsanity.com/paper/1910.06711/) (2019)
- [cGANs with Projection Discriminator](https://mlsanity.com/paper/1802.05637/) (2018)
- [BigVGAN: A Universal Neural Vocoder with Large-Scale Training](https://mlsanity.com/paper/2206.04658/) (2022)
- [Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders](https://mlsanity.com/paper/1704.01279/) (2017)
- [DDSP: Differentiable Digital Signal Processing](https://mlsanity.com/paper/2001.04643/) (2020)
- [Instance-Conditioned GAN](https://mlsanity.com/paper/2109.05070/) (2021)
- [Fader Networks: Manipulating Images by Sliding Attributes](https://mlsanity.com/paper/1706.00409/) (2017)
- [GANSynth: Adversarial Neural Audio Synthesis](https://mlsanity.com/paper/1902.08710/) (2019)
- [TorchAudio: Building Blocks for Audio and Speech Processing](https://mlsanity.com/paper/2110.15018/) (2021)
- [MelNet: A Generative Model for Audio in the Frequency Domain](https://mlsanity.com/paper/1906.01083/) (2019)
- [DrumGAN: Synthesis of Drum Sounds With Timbral Feature Conditioning Using Generative Adversarial Networks](https://mlsanity.com/paper/2008.12073/) (2020)
- [A Universal Music Translation Network](https://mlsanity.com/paper/1805.07848/) (2018)
- [Learning Disentangled Representations of Timbre and Pitch for Musical Instrument Sounds Using Gaussian Mixture Variational Autoencoders](https://mlsanity.com/paper/1906.08152/) (2019)
- [DDSP-based Singing Vocoders: A New Subtractive-based Synthesizer and A Comprehensive Evaluation](https://mlsanity.com/paper/2208.04756/) (2022)
- [Differentiable Wavetable Synthesis](https://mlsanity.com/paper/2111.10003/) (2021)

## Built on by (most cited first, 9 of 9)

- [Expressive Acoustic Guitar Sound Synthesis with an Instrument-Specific Input Representation and Diffusion Outpainting](https://mlsanity.com/paper/2401.13498/) (2024)
- [TokenSynth: A Token-based Neural Synthesizer for Instrument Cloning and Text-to-Instrument](https://mlsanity.com/paper/2502.08939/) (2025)
- [InstrumentGen: Generating Sample-Based Musical Instruments From Text](https://mlsanity.com/paper/2311.04339/) (2023)
- [Annotation-free Automatic Music Transcription with Scalable Synthetic Data and Adversarial Domain Confusion](https://mlsanity.com/paper/2312.10402/) (2023)
- [HyperGANStrument: Instrument Sound Synthesis and Editing with Pitch-Invariant Hypernetworks](https://mlsanity.com/paper/2401.04558/) (2024)
- [Generating Sample-Based Musical Instruments Using Neural Audio Codec Language Models](https://mlsanity.com/paper/2407.15641/) (2024)
- [Automatic Equalization for Individual Instrument Tracks Using Convolutional Neural Networks](https://mlsanity.com/paper/2407.16691/) (2024)
- [Designing Neural Synthesizers for Low-Latency Interaction](https://mlsanity.com/paper/2503.11562/) (2025)
- [Pitch-Conditioned Instrument Sound Synthesis From an Interactive Timbre Latent Space](https://mlsanity.com/paper/2510.04339/) (2025)

## Abstract (arXiv)

We propose GANStrument, a generative adversarial model for instrument sound synthesis. Given a one-shot sound as input, it is able to generate pitched instrument sounds that reflect the timbre of the input within an interactive time. By exploiting instance conditioning, GANStrument achieves better fidelity and diversity of synthesized sounds and generalization ability to various inputs. In addition, we introduce an adversarial training scheme for a pitch-invariant feature extractor that significantly improves the pitch accuracy and timbre consistency. Experimental results show that GANStrument outperforms strong baselines that do not use instance conditioning in terms of generation quality and input editability. Qualitative examples are available online.