# HyperGANStrument: Instrument Sound Synthesis and Editing with Pitch-Invariant Hypernetworks

arXiv: https://arxiv.org/abs/2401.04558 · HTML: https://mlsanity.com/paper/2401.04558/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 24 references that are in the map.
- First posted on arXiv 9 Jan 2024.

## Builds on (most cited first, 19 of 19)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Analyzing and Improving the Image Quality of StyleGAN](https://mlsanity.com/paper/1912.04958/) (2019)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Generative Visual Manipulation on the Natural Image Manifold](https://mlsanity.com/paper/1609.03552/) (2016)
- [cGANs with Projection Discriminator](https://mlsanity.com/paper/1802.05637/) (2018)
- [ReStyle: A Residual-Based StyleGAN Encoder via Iterative Refinement](https://mlsanity.com/paper/2104.02699/) (2021)
- [Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders](https://mlsanity.com/paper/1704.01279/) (2017)
- [HyperStyle: StyleGAN Inversion with HyperNetworks for Real Image Editing](https://mlsanity.com/paper/2111.15666/) (2021)
- [High-Fidelity GAN Inversion for Image Attribute Editing](https://mlsanity.com/paper/2109.06590/) (2021)
- [DDSP: Differentiable Digital Signal Processing](https://mlsanity.com/paper/2001.04643/) (2020)
- [Instance-Conditioned GAN](https://mlsanity.com/paper/2109.05070/) (2021)
- [GANSynth: Adversarial Neural Audio Synthesis](https://mlsanity.com/paper/1902.08710/) (2019)
- [HyperInverter: Improving StyleGAN Inversion via Hypernetwork](https://mlsanity.com/paper/2112.00719/) (2021)
- [RAVE: A variational autoencoder for fast and high-quality neural audio synthesis](https://mlsanity.com/paper/2111.05011/) (2021)
- [DrumGAN: Synthesis of Drum Sounds With Timbral Feature Conditioning Using Generative Adversarial Networks](https://mlsanity.com/paper/2008.12073/) (2020)
- [Learning Disentangled Representations of Timbre and Pitch for Musical Instrument Sounds Using Gaussian Mixture Variational Autoencoders](https://mlsanity.com/paper/1906.08152/) (2019)
- [GANStrument: Adversarial Instrument Sound Synthesis with Pitch-invariant Instance Conditioning](https://mlsanity.com/paper/2211.05385/) (2022)
- [Differentiable Wavetable Synthesis](https://mlsanity.com/paper/2111.10003/) (2021)

## Built on by (most cited first, 1 of 1)

- [Pitch-Conditioned Instrument Sound Synthesis From an Interactive Timbre Latent Space](https://mlsanity.com/paper/2510.04339/) (2025)

## Abstract (arXiv)

GANStrument, exploiting GANs with a pitch-invariant feature extractor and instance conditioning technique, has shown remarkable capabilities in synthesizing realistic instrument sounds. To further improve the reconstruction ability and pitch accuracy to enhance the editability of user-provided sound, we propose HyperGANStrument, which introduces a pitch-invariant hypernetwork to modulate the weights of a pre-trained GANStrument generator, given a one-shot sound as input. The hypernetwork modulation provides feedback for the generator in the reconstruction of the input sound. In addition, we take advantage of an adversarial fine-tuning scheme for the hypernetwork to improve the reconstruction fidelity and generation diversity of the generator. Experimental results show that the proposed model not only enhances the generation capability of GANStrument but also significantly improves the editability of synthesized sounds. Audio examples are available at the online demo page.