# Adversarial Generation of Time-Frequency Features with application in audio synthesis

arXiv: https://arxiv.org/abs/1902.04072 · HTML: https://mlsanity.com/paper/1902.04072/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Feb 2019.

## Built on by (most cited first, 13 of 13)

- [FoleyGAN: Visually Guided Generative Adversarial Network-Based Synchronous Sound Generation in Silent Videos](https://mlsanity.com/paper/2107.09262/) (2021)
- [Catch-A-Waveform: Learning to Generate Audio from a Single Short Example](https://mlsanity.com/paper/2106.06426/) (2021)
- [Synthesis of standard 12-lead electrocardiograms using two dimensional generative adversarial network](https://mlsanity.com/paper/2106.03701/) (2021)
- [Real-Time Neural Voice Camouflage](https://mlsanity.com/paper/2112.07076/) (2021)
- [Msanii: High Fidelity Music Synthesis on a Shoestring Budget](https://mlsanity.com/paper/2301.06468/) (2023)
- [A Survey of Deep Learning Audio Generation Methods](https://mlsanity.com/paper/2406.00146/) (2024)
- [Adversarial Audio Synthesis with Complex-valued Polynomial Networks](https://mlsanity.com/paper/2206.06811/) (2022)
- [Data-Driven Modeling of Noise Time Series with Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/2207.01110/) (2022)
- [Example-Based Framework for Perceptually Guided Audio Texture Generation](https://mlsanity.com/paper/2308.11859/) (2023)
- [ReCon: Reconfiguring Analog Rydberg Atom Quantum Computers for Quantum Generative Adversarial Networks](https://mlsanity.com/paper/2408.13389/) (2024)
- [Unified Signal Compression Using a GAN with Iterative Latent Representation Optimization](https://mlsanity.com/paper/2109.11168/) (2021)
- [OrganiQ: Mitigating Classical Resource Bottlenecks of Quantum Generative Adversarial Networks on NISQ-Era Machines](https://mlsanity.com/paper/2409.19823/) (2024)
- [Stage-Wise and Prior-Aware Neural Speech Phase Prediction](https://mlsanity.com/paper/2410.04990/) (2024)

## Abstract (arXiv)

Time-frequency (TF) representations provide powerful and intuitive features for the analysis of time series such as audio. But still, generative modeling of audio in the TF domain is a subtle matter. Consequently, neural audio synthesis widely relies on directly modeling the waveform and previous attempts at unconditionally synthesizing audio from neurally generated invertible TF features still struggle to produce audio at satisfying quality. In this article, focusing on the short-time Fourier transform, we discuss the challenges that arise in audio synthesis based on generated invertible TF features and how to overcome them. We demonstrate the potential of deliberate generative TF modeling by training a generative adversarial network (GAN) on short-time Fourier features. We show that by applying our guidelines, our TF-based network was able to outperform a state-of-the-art GAN generating waveforms directly, despite the similar architecture in the two networks.