# Evaluating generative audio systems and their metrics

arXiv: https://arxiv.org/abs/2209.00130 · HTML: https://mlsanity.com/paper/2209.00130/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 30 references that are in the map.
- First posted on arXiv 31 Aug 2022.

## Builds on (most cited first, 17 of 17)

- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [DiffWave: A Versatile Diffusion Model for Audio Synthesis](https://mlsanity.com/paper/2009.09761/) (2020)
- [Demystifying MMD GANs](https://mlsanity.com/paper/1801.01401/) (2018)
- [Tacotron: Towards End-to-End Speech Synthesis](https://mlsanity.com/paper/1703.10135/) (2017)
- [WaveGlow: A Flow-based Generative Network for Speech Synthesis](https://mlsanity.com/paper/1811.00002/) (2018)
- [A note on the evaluation of generative models](https://mlsanity.com/paper/1511.01844/) (2015)
- [Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders](https://mlsanity.com/paper/1704.01279/) (2017)
- [Adversarial Audio Synthesis](https://mlsanity.com/paper/1802.04208/) (2018)
- [DDSP: Differentiable Digital Signal Processing](https://mlsanity.com/paper/2001.04643/) (2020)
- [GANSynth: Adversarial Neural Audio Synthesis](https://mlsanity.com/paper/1902.08710/) (2019)
- [On GANs and GMMs](https://mlsanity.com/paper/1805.12462/) (2018)
- [CRASH: Raw Audio Score-based Generative Modeling for Controllable High-resolution Drum Sound Synthesis](https://mlsanity.com/paper/2106.07431/) (2021)
- [Comparing Representations for Audio Synthesis Using Generative Adversarial Networks](https://mlsanity.com/paper/2006.09266/) (2020)
- [Neural Waveshaping Synthesis](https://mlsanity.com/paper/2107.05050/) (2021)
- [DarkGAN: Exploiting Knowledge Distillation for Comprehensible Audio Synthesis with GANs](https://mlsanity.com/paper/2108.01216/) (2021)
- [VQCPC-GAN: Variable-Length Adversarial Audio Synthesis Using Vector-Quantized Contrastive Predictive Coding](https://mlsanity.com/paper/2105.01531/) (2021)

## Built on by (most cited first, 16 of 16)

- [Multi-Source Diffusion Models for Simultaneous Music Generation and Separation](https://mlsanity.com/paper/2302.02257/) (2023)
- [Aligning Text-to-Music Evaluation with Human Preferences](https://mlsanity.com/paper/2503.16669/) (2025)
- [Benchmarking Music Generation Models and Metrics via Human Preference Studies](https://mlsanity.com/paper/2506.19085/) (2025)
- [Siamese SIREN: Audio Compression with Implicit Neural Representations](https://mlsanity.com/paper/2306.12957/) (2023)
- [Survey on the Evaluation of Generative Models in Music](https://mlsanity.com/paper/2506.05104/) (2025)
- [Multi-modal Latent Diffusion](https://mlsanity.com/paper/2306.04445/) (2023)
- [Subtractive Training for Music Stem Insertion using Latent Diffusion Models](https://mlsanity.com/paper/2406.19328/) (2024)
- [Benchmarking Single-Factor Physical Video-to-Audio Generation](https://mlsanity.com/paper/2605.30339/) (2026)
- [What You Hear Is What You See: Audio Quality Metrics From Image Quality Metrics](https://mlsanity.com/paper/2305.11582/) (2023)
- [Data is Overrated: Perceptual Metrics Can Lead Learning in the Absence of Training Data](https://mlsanity.com/paper/2312.03455/) (2023)
- [Latent CLAP Loss for Better Foley Sound Synthesis](https://mlsanity.com/paper/2403.12182/) (2024)
- [Prevailing Research Areas for Music AI in the Era of Foundation Models](https://mlsanity.com/paper/2409.09378/) (2024)
- [Source Separation of Multi-source Raw Music using a Residual Quantized Variational Autoencoder](https://mlsanity.com/paper/2408.07020/) (2024)
- [The Effect of Perceptual Metrics on Music Representation Learning for Genre Classification](https://mlsanity.com/paper/2409.17069/) (2024)
- [MARS: Sound Generation via Multi-Channel Autoregression on Spectrograms](https://mlsanity.com/paper/2509.26007/) (2025)
- [Story2MIDI: Emotionally Aligned Music Generation from Text](https://mlsanity.com/paper/2512.02192/) (2025)

## Abstract (arXiv)

Recent years have seen considerable advances in audio synthesis with deep generative models. However, the state-of-the-art is very difficult to quantify; different studies often use different evaluation methodologies and different metrics when reporting results, making a direct comparison to other systems difficult if not impossible. Furthermore, the perceptual relevance and meaning of the reported metrics in most cases unknown, prohibiting any conclusive insights with respect to practical usability and audio quality. This paper presents a study that investigates state-of-the-art approaches side-by-side with (i) a set of previously proposed objective metrics for audio reconstruction, and with (ii) a listening study. The results indicate that currently used objective metrics are insufficient to describe the perceptual quality of current systems.