# CDPAM: Contrastive learning for perceptual audio similarity

arXiv: https://arxiv.org/abs/2102.05109 · HTML: https://mlsanity.com/paper/2102.05109/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 40 references that are in the map.
- First posted on arXiv 9 Feb 2021.

## Builds on (most cited first, 23 of 23)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Unsupervised Cross-lingual Representation Learning for Speech Recognition](https://mlsanity.com/paper/2006.13979/) (2020)
- [MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis](https://mlsanity.com/paper/1910.06711/) (2019)
- [Towards a Definition of Disentangled Representations](https://mlsanity.com/paper/1812.02230/) (2018)
- [Real Time Speech Enhancement in the Waveform Domain](https://mlsanity.com/paper/2006.12847/) (2020)
- [MOSNet: Deep Learning based Objective Assessment for Voice Conversion](https://mlsanity.com/paper/1904.08352/) (2019)
- [MetricGAN: Generative Adversarial Networks based Black-box Metric Scores Optimization for Speech Enhancement](https://mlsanity.com/paper/1905.04874/) (2019)
- [Are Disentangled Representations Helpful for Abstract Visual Reasoning?](https://mlsanity.com/paper/1905.12506/) (2019)
- [The Voice Conversion Challenge 2018: Promoting Development of Parallel and Nonparallel Methods](https://mlsanity.com/paper/1804.04262/) (2018)
- [Audio ALBERT: A Lite BERT for Self-supervised Learning of Audio Representation](https://mlsanity.com/paper/2005.08575/) (2020)
- [Self-Supervised Contrastive Learning for Unsupervised Phoneme Segmentation](https://mlsanity.com/paper/2007.13465/) (2020)
- [HiFi-GAN: High-Fidelity Denoising and Dereverberation Based on Speech Deep Features in Adversarial Networks](https://mlsanity.com/paper/2006.05694/) (2020)
- [Multi-target Voice Conversion without Parallel Data by Adversarially Learning Disentangled Audio Representations](https://mlsanity.com/paper/1804.02812/) (2018)
- [A Differentiable Perceptual Audio Metric Learned from Just Noticeable Differences](https://mlsanity.com/paper/2001.04460/) (2020)
- [AutoMOS: Learning a non-intrusive assessor of naturalness-of-speech](https://mlsanity.com/paper/1611.09207/) (2016)
- [Disentangled Multidimensional Metric Learning for Music Similarity](https://mlsanity.com/paper/2008.03720/) (2020)
- [SESQA: semi-supervised learning for speech quality assessment](https://mlsanity.com/paper/2010.00368/) (2020)
- [The INTERSPEECH 2020 Deep Noise Suppression Challenge: Datasets, Subjective Speech Quality and Testing Framework](https://mlsanity.com/paper/2001.08662/) (2020)
- [Learning with Learned Loss Function: Speech Enhancement with Quality-Net to Improve Perceptual Evaluation of Speech Quality](https://mlsanity.com/paper/1905.01898/) (2019)
- [Content-based Representations of audio using Siamese neural networks](https://mlsanity.com/paper/1710.10974/) (2017)
- [Learning Disentangled Representations for Timber and Pitch in Music Audio](https://mlsanity.com/paper/1811.03271/) (2018)

## Built on by (most cited first, 30 of 42)

- [DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data](https://mlsanity.com/paper/2306.09344/) (2023)
- [The ObjectFolder Benchmark: Multisensory Learning with Neural and Real Objects](https://mlsanity.com/paper/2306.00956/) (2023)
- [Self-supervised Contrastive Cross-Modality Representation Learning for Spoken Question Answering](https://mlsanity.com/paper/2109.03381/) (2021)
- [Non-Contrastive Learning Meets Language-Image Pre-Training](https://mlsanity.com/paper/2210.09304/) (2022)
- [KAD: No More FAD! An Effective and Efficient Evaluation Metric for Audio Generation](https://mlsanity.com/paper/2502.15602/) (2025)
- [SCoFT: Self-Contrastive Fine-Tuning for Equitable Image Generation](https://mlsanity.com/paper/2401.08053/) (2024)
- [CALICO: Self-Supervised Camera-LiDAR Contrastive Pre-training for BEV Perception](https://mlsanity.com/paper/2306.00349/) (2023)
- [Siamese SIREN: Audio Compression with Implicit Neural Representations](https://mlsanity.com/paper/2306.12957/) (2023)
- [HyperSound: Generating Implicit Neural Representations of Audio Signals with Hypernetworks](https://mlsanity.com/paper/2211.01839/) (2022)
- [DPM-TSE: A Diffusion Probabilistic Model for Target Sound Extraction](https://mlsanity.com/paper/2310.04567/) (2023)
- [Music Enhancement via Image Translation and Vocoding](https://mlsanity.com/paper/2204.13289/) (2022)
- [Hypernetworks build Implicit Neural Representations of Sounds](https://mlsanity.com/paper/2302.04959/) (2023)
- [Self-supervised Context-aware Style Representation for Expressive Speech Synthesis](https://mlsanity.com/paper/2206.12559/) (2022)
- [Equivariant Self-Supervision for Musical Tempo Estimation](https://mlsanity.com/paper/2209.01478/) (2022)
- [RealImpact: A Dataset of Impact Sound Fields for Real Objects](https://mlsanity.com/paper/2306.09944/) (2023)
- [Synthio: Augmenting Small-Scale Audio Classification Datasets with Synthetic Data](https://mlsanity.com/paper/2410.02056/) (2024)
- [FINALLY: fast and universal speech enhancement with studio-like quality](https://mlsanity.com/paper/2410.05920/) (2024)
- [Towards Lightweight Controllable Audio Synthesis with Conditional Implicit Neural Representations](https://mlsanity.com/paper/2111.08462/) (2021)
- [Contrastive encoder pre-training-based clustered federated learning for heterogeneous data](https://mlsanity.com/paper/2311.16535/) (2023)
- [SoundMorpher: Perceptually-Uniform Sound Morphing with Diffusion Model](https://mlsanity.com/paper/2410.02144/) (2024)
- [High-Fidelity Music Vocoder using Neural Audio Codecs](https://mlsanity.com/paper/2502.12759/) (2025)
- [Reproducible Subjective Evaluation](https://mlsanity.com/paper/2203.04444/) (2022)
- [Efficient Speech Quality Assessment using Self-supervised Framewise Embeddings](https://mlsanity.com/paper/2211.06646/) (2022)
- [Towards Robust FastSpeech 2 by Modelling Residual Multimodality](https://mlsanity.com/paper/2306.01442/) (2023)
- [Machine Perceptual Quality: Evaluating the Impact of Severe Lossy Compression on Audio and Image Models](https://mlsanity.com/paper/2401.07957/) (2024)
- [Learned Compression for Compressed Learning](https://mlsanity.com/paper/2412.09405/) (2024)
- [Hearing Hands: Generating Sounds from Physical Interactions in 3D Scenes](https://mlsanity.com/paper/2506.09989/) (2025)
- [Predicting pairwise preferences between TTS audio stimuli using parallel ratings data and anti-symmetric twin neural networks](https://mlsanity.com/paper/2209.11003/) (2022)
- [ViFiCon: Vision and Wireless Association Via Self-Supervised Contrastive Learning](https://mlsanity.com/paper/2210.05513/) (2022)
- [What You Hear Is What You See: Audio Quality Metrics From Image Quality Metrics](https://mlsanity.com/paper/2305.11582/) (2023)

## Abstract (arXiv)

Many speech processing methods based on deep learning require an automatic and differentiable audio metric for the loss function. The DPAM approach of Manocha et al. learns a full-reference metric trained directly on human judgments, and thus correlates well with human perception. However, it requires a large number of human annotations and does not generalize well outside the range of perturbations on which it was trained. This paper introduces CDPAM, a metric that builds on and advances DPAM. The primary improvement is to combine contrastive learning and multi-dimensional representations to build robust models from limited data. In addition, we collect human judgments on triplet comparisons to improve generalization to a broader range of audio perturbations. CDPAM correlates well with human responses across nine varied datasets. We also show that adding this metric to existing speech synthesis and enhancement methods yields significant improvement, as measured by objective and subjective tests.