# Music Enhancement via Image Translation and Vocoding

arXiv: https://arxiv.org/abs/2204.13289 · HTML: https://mlsanity.com/paper/2204.13289/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 32 references that are in the map.
- First posted on arXiv 28 Apr 2022.

## Builds on (most cited first, 21 of 21)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [DiffWave: A Versatile Diffusion Model for Audio Synthesis](https://mlsanity.com/paper/2009.09761/) (2020)
- [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016)
- [WaveGrad: Estimating Gradients for Waveform Generation](https://mlsanity.com/paper/2009.00713/) (2020)
- [Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation](https://mlsanity.com/paper/1809.07454/) (2018)
- [MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis](https://mlsanity.com/paper/1910.06711/) (2019)
- [Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram](https://mlsanity.com/paper/1910.11480/) (2019)
- [DNSMOS: A Non-Intrusive Perceptual Objective Speech Quality metric to evaluate Noise Suppressors](https://mlsanity.com/paper/2010.15258/) (2020)
- [Fr'echet Audio Distance: A Metric for Evaluating Music Enhancement Algorithms](https://mlsanity.com/paper/1812.08466/) (2018)
- [Real Time Speech Enhancement in the Waveform Domain](https://mlsanity.com/paper/2006.12847/) (2020)
- [Music Source Separation in the Waveform Domain](https://mlsanity.com/paper/1911.13254/) (2019)
- [TorchAudio: Building Blocks for Audio and Speech Processing](https://mlsanity.com/paper/2110.15018/) (2021)
- [Interspeech 2021 Deep Noise Suppression Challenge](https://mlsanity.com/paper/2101.01902/) (2021)
- [CDPAM: Contrastive learning for perceptual audio similarity](https://mlsanity.com/paper/2102.05109/) (2021)
- [HiFi-GAN: High-Fidelity Denoising and Dereverberation Based on Speech Deep Features in Adversarial Networks](https://mlsanity.com/paper/2006.05694/) (2020)
- [GAN Vocoder: Multi-Resolution Discriminator Is All You Need](https://mlsanity.com/paper/2103.05236/) (2021)
- [Deep convolutional networks on the pitch spiral for musical instrument recognition](https://mlsanity.com/paper/1605.06644/) (2016)
- [High Fidelity Speech Regeneration with Application to Speech Enhancement](https://mlsanity.com/paper/2102.00429/) (2021)
- [Towards Generalized Speech Enhancement with Generative Adversarial Networks](https://mlsanity.com/paper/1904.03418/) (2019)

## Built on by (most cited first, 11 of 11)

- [GibbsDDRM: A Partially Collapsed Gibbs Sampler for Solving Blind Inverse Problems with Denoising Diffusion Restoration](https://mlsanity.com/paper/2301.12686/) (2023)
- [Multi-instrument Music Synthesis with Spectrogram Diffusion](https://mlsanity.com/paper/2206.05408/) (2022)
- [MaskSR: Masked Language Model for Full-band Speech Restoration](https://mlsanity.com/paper/2406.02092/) (2024)
- [Unsupervised vocal dereverberation with diffusion-based generative models](https://mlsanity.com/paper/2211.04124/) (2022)
- [Unsupervised Blind Joint Dereverberation and Room Acoustics Estimation with Diffusion Models](https://mlsanity.com/paper/2408.07472/) (2024)
- [Music Separation Enhancement with Generative Modeling](https://mlsanity.com/paper/2208.12387/) (2022)
- [SonicMaster: Towards Controllable All-in-One Music Restoration and Mastering](https://mlsanity.com/paper/2508.03448/) (2025)
- [Mel Spectrogram Inversion with Stable Pitch](https://mlsanity.com/paper/2208.12782/) (2022)
- [Exploiting Time-Frequency Conformers for Music Audio Enhancement](https://mlsanity.com/paper/2308.12599/) (2023)
- [Audio Denoising for Robust Audio Fingerprinting](https://mlsanity.com/paper/2212.11277/) (2022)
- [Music Augmentation and Denoising For Peak-Based Audio Fingerprinting](https://mlsanity.com/paper/2310.13388/) (2023)

## Abstract (arXiv)

Consumer-grade music recordings such as those captured by mobile devices typically contain distortions in the form of background noise, reverb, and microphone-induced EQ. This paper presents a deep learning approach to enhance low-quality music recordings by combining (i) an image-to-image translation model for manipulating audio in its mel-spectrogram representation and (ii) a music vocoding model for mapping synthetically generated mel-spectrograms to perceptually realistic waveforms. We find that this approach to music enhancement outperforms baselines which use classical methods for mel-spectrogram inversion and an end-to-end approach directly mapping noisy waveforms to clean waveforms. Additionally, in evaluating the proposed method with a listening test, we analyze the reliability of common audio enhancement evaluation metrics when used in the music domain.