mlsanityData to 7 Oct 2026

CDPAM: Contrastive learning for perceptual audio similarity

arXiv 2102.05109 ↗First posted 9 Feb 2021eess.AS cs.LG cs.SDcited by 42

Not on the map. Why?

Abstract (arXiv)

Many speech processing methods based on deep learning require an automatic and differentiable audio metric for the loss function. The DPAM approach of Manocha et al. learns a full-reference metric trained directly on human judgments, and thus correlates well with human perception. However, it requires a large number of human annotations and does not generalize well outside the range of perturbations on which it was trained. This paper introduces CDPAM, a metric that builds on and advances DPAM. The primary improvement is to combine contrastive learning and multi-dimensional representations to build robust models from limited data. In addition, we collect human judgments on triplet comparisons to improve generalization to a broader range of audio perturbations. CDPAM correlates well with human responses across nine varied datasets. We also show that adding this metric to existing speech synthesis and enhancement methods yields significant improvement, as measured by objective and subjective tests.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Builds on 23 of its 40 references that are in the map. First posted on arXiv 9 Feb 2021.

Builds on

23 of 40 references.

References · 23

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 496
  5. 2019MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesiscited by 351
  6. 2018Towards a Definition of Disentangled Representationscited by 294
  7. 2020Real Time Speech Enhancement in the Waveform Domaincited by 183
  8. 2019MOSNet: Deep Learning based Objective Assessment for Voice Conversioncited by 126
  9. 2019MetricGAN: Generative Adversarial Networks based Black-box Metric Scores Optimization for Speech Enhancementcited by 104
  10. 2019Are Disentangled Representations Helpful for Abstract Visual Reasoning?cited by 100
  11. 2018The Voice Conversion Challenge 2018: Promoting Development of Parallel and Nonparallel Methodscited by 68
  12. 2020Audio ALBERT: A Lite BERT for Self-supervised Learning of Audio Representationcited by 58
11 more
  1. 2020Self-Supervised Contrastive Learning for Unsupervised Phoneme Segmentationcited by 43
  2. 2020HiFi-GAN: High-Fidelity Denoising and Dereverberation Based on Speech Deep Features in Adversarial Networkscited by 40
  3. 2018Multi-target Voice Conversion without Parallel Data by Adversarially Learning Disentangled Audio Representationscited by 25
  4. 2016AutoMOS: Learning a non-intrusive assessor of naturalness-of-speechcited by 22
  5. 2020A Differentiable Perceptual Audio Metric Learned from Just Noticeable Differencescited by 22
  6. 2020Disentangled Multidimensional Metric Learning for Music Similaritycited by 16
  7. 2020SESQA: semi-supervised learning for speech quality assessmentcited by 13
  8. 2020The INTERSPEECH 2020 Deep Noise Suppression Challenge: Datasets, Subjective Speech Quality and Testing Frameworkcited by 11
  9. 2019Learning with Learned Loss Function: Speech Enhancement with Quality-Net to Improve Perceptual Evaluation of Speech Qualitycited by 8
  10. 2017Content-based Representations of audio using Siamese neural networkscited by 4
  11. 2018Learning Disentangled Representations for Timber and Pitch in Music Audiocited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 9; 2023: 12; 2024: 8; 2025: 6; 2026: 5.

42 papers.

Sort:

By area:

  1. 2023DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Datacited by 322
  2. 2023The ObjectFolder Benchmark: Multisensory Learning with Neural and Real Objectscited by 40
  3. 2021Self-supervised Contrastive Cross-Modality Representation Learning for Spoken Question Answeringcited by 39
  4. 2022Non-Contrastive Learning Meets Language-Image Pre-Trainingcited by 25
  5. 2025KAD: No More FAD! An Effective and Efficient Evaluation Metric for Audio Generationcited by 24
  6. 2024SCoFT: Self-Contrastive Fine-Tuning for Equitable Image Generationcited by 19
  7. 2023CALICO: Self-Supervised Camera-LiDAR Contrastive Pre-training for BEV Perceptioncited by 14
  8. 2022HyperSound: Generating Implicit Neural Representations of Audio Signals with Hypernetworkscited by 12
  9. 2023DPM-TSE: A Diffusion Probabilistic Model for Target Sound Extractioncited by 12
  10. 2022Music Enhancement via Image Translation and Vocodingcited by 11
  11. 2023Hypernetworks build Implicit Neural Representations of Soundscited by 11
  12. 2023Siamese SIREN: Audio Compression with Implicit Neural Representationscited by 11
30 more
  1. 2022Self-supervised Context-aware Style Representation for Expressive Speech Synthesiscited by 10
  2. 2022Equivariant Self-Supervision for Musical Tempo Estimationcited by 9
  3. 2023RealImpact: A Dataset of Impact Sound Fields for Real Objectscited by 8
  4. 2024Synthio: Augmenting Small-Scale Audio Classification Datasets with Synthetic Datacited by 7
  5. 2024FINALLY: fast and universal speech enhancement with studio-like qualitycited by 7
  6. 2021Towards Lightweight Controllable Audio Synthesis with Conditional Implicit Neural Representationscited by 5
  7. 2023Contrastive encoder pre-training-based clustered federated learning for heterogeneous datacited by 5
  8. 2024SoundMorpher: Perceptually-Uniform Sound Morphing with Diffusion Modelcited by 3
  9. 2022Reproducible Subjective Evaluationcited by 2
  10. 2022Efficient Speech Quality Assessment using Self-supervised Framewise Embeddingscited by 2
  11. 2023Towards Robust FastSpeech 2 by Modelling Residual Multimodalitycited by 2
  12. 2024Machine Perceptual Quality: Evaluating the Impact of Severe Lossy Compression on Audio and Image Modelscited by 2
  13. 2024Learned Compression for Compressed Learningcited by 2
  14. 2025High-Fidelity Music Vocoder using Neural Audio Codecscited by 2
  15. 2025Hearing Hands: Generating Sounds from Physical Interactions in 3D Scenescited by 2
  16. 2022Predicting pairwise preferences between TTS audio stimuli using parallel ratings data and anti-symmetric twin neural networkscited by 1
  17. 2022ViFiCon: Vision and Wireless Association Via Self-Supervised Contrastive Learningcited by 1
  18. 2023What You Hear Is What You See: Audio Quality Metrics From Image Quality Metricscited by 1
  19. 2023CORN: Co-Trained Full- And No-Reference Speech Quality Assessmentcited by 1
  20. 2024Contrastive Learning from Synthetic Audio Doppelg"angerscited by 1
  21. 2025JSQA: Speech Quality Assessment with Perceptually-Inspired Contrastive Pretraining Based on JND Audio Pairscited by 1
  22. 2026SAM Audio Judge: A Unified Multimodal Framework for Perceptual Evaluation of Audio Separationcited by 1
  23. 2026PHALAR: Phasors for Learned Musical Audio Representationscited by 1
  24. 2023Self-supervised Auxiliary Loss for Metric Learning in Music Similarity-based Retrieval and Auto-taggingcited by 0
  25. 2024Addressing Emotion Bias in Music Emotion Recognition and Generation with Frechet Audio Distancecited by 0
  26. 2025As Good as It KAN Get: High-Fidelity Audio Representationcited by 0
  27. 2025MMMOS: Multi-domain Multi-axis Audio Quality Assessmentcited by 0
  28. 2026The Perceived Fragility of Explanations in Audio Models: Manipulation of Attribution with Unchanged Predictionscited by 0
  29. 2026Objects as Audio-Visual Modal Sound Fieldscited by 0
  30. 2026Audio-Driven Adversarial Defense for 3D Talking Face Generation with totally Visual Fidelity Preservationcited by 0