# Learning Repeatable Speech Embeddings Using An Intra-class Correlation Regularizer

arXiv: https://arxiv.org/abs/2310.17049 · HTML: https://mlsanity.com/paper/2310.17049/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 62 references that are in the map.
- First posted on arXiv 25 Oct 2023.

## Builds on (most cited first, 30 of 35)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning](https://mlsanity.com/paper/2105.04906/) (2021)
- [VoxCeleb2: Deep Speaker Recognition](https://mlsanity.com/paper/1806.05622/) (2018)
- [What Makes for Good Views for Contrastive Learning?](https://mlsanity.com/paper/2005.10243/) (2020)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [A Closer Look at Few-shot Classification](https://mlsanity.com/paper/1904.04232/) (2019)
- [Generalized End-to-End Loss for Speaker Verification](https://mlsanity.com/paper/1710.10467/) (2017)
- [AUTOVC: Zero-Shot Voice Style Transfer with Only Autoencoder Loss](https://mlsanity.com/paper/1905.05879/) (2019)
- [FSCE: Few-Shot Object Detection via Contrastive Proposal Encoding](https://mlsanity.com/paper/2103.05950/) (2021)
- [In defence of metric learning for speaker recognition](https://mlsanity.com/paper/2003.11982/) (2020)
- [Deep Speaker: an End-to-End Neural Speaker Embedding System](https://mlsanity.com/paper/1705.02304/) (2017)
- [Exploring Complementary Strengths of Invariant and Equivariant Representations for Few-Shot Learning](https://mlsanity.com/paper/2103.01315/) (2021)
- [End-to-End Text-Dependent Speaker Verification](https://mlsanity.com/paper/1509.08062/) (2015)
- [Pose Invariant Embedding for Deep Person Re-identification](https://mlsanity.com/paper/1701.07732/) (2017)
- [View-Invariant Probabilistic Embedding for Human Pose](https://mlsanity.com/paper/1912.01001/) (2019)
- [A Theoretical Analysis of the Number of Shots in Few-Shot Learning](https://mlsanity.com/paper/1909.11722/) (2019)
- [Learning Compatible Embeddings](https://mlsanity.com/paper/2108.01958/) (2021)
- [RawNet: Advanced end-to-end deep neural network using raw waveforms for text-independent speaker verification](https://mlsanity.com/paper/1904.08104/) (2019)
- [Robust Re-identification of Manta Rays from Natural Markings by Learning Pose Invariant Embeddings](https://mlsanity.com/paper/1902.10847/) (2019)
- [How Far Are We from Robust Voice Conversion: A Survey](https://mlsanity.com/paper/2011.12063/) (2020)
- [Apparel-invariant Feature Learning for Apparel-changed Person Re-identification](https://mlsanity.com/paper/2008.06181/) (2020)
- [Deep Speaker Embedding Learning with Multi-Level Pooling for Text-Independent Speaker Verification](https://mlsanity.com/paper/1902.07821/) (2019)
- [Voice Pathology Detection Using Deep Learning: a Preliminary Study](https://mlsanity.com/paper/1907.05905/) (2019)
- [Within-sample variability-invariant loss for robust speaker recognition under noisy environments](https://mlsanity.com/paper/2002.00924/) (2020)
- [Intra-class variation reduction of speaker representation in disentanglement framework](https://mlsanity.com/paper/2008.01348/) (2020)
- [DGC-vector: A new speaker embedding for zero-shot voice conversion](https://mlsanity.com/paper/2203.09722/) (2022)
- [On Intra-Class Variance for Deep Learning of Classifiers](https://mlsanity.com/paper/1901.11186/) (2019)

## Built on by (most cited first, 3 of 3)

- [From Faces to Voices: Learning Hierarchical Representations for High-quality Video-to-Speech](https://mlsanity.com/paper/2503.16956/) (2025)
- [A Tutorial on Clinical Speech AI Development: From Data Collection to Model Validation](https://mlsanity.com/paper/2410.21640/) (2024)
- [Rethinking Speaker Embeddings for Speech Generation: Sub-Center Modeling for Capturing Intra-Speaker Diversity](https://mlsanity.com/paper/2407.04291/) (2024)

## Abstract (arXiv)

A good supervised embedding for a specific machine learning task is only sensitive to changes in the label of interest and is invariant to other confounding factors. We leverage the concept of repeatability from measurement theory to describe this property and propose to use the intra-class correlation coefficient (ICC) to evaluate the repeatability of embeddings. We then propose a novel regularizer, the ICC regularizer, as a complementary component for contrastive losses to guide deep neural networks to produce embeddings with higher repeatability. We use simulated data to explain why the ICC regularizer works better on minimizing the intra-class variance than the contrastive loss alone. We implement the ICC regularizer and apply it to three speech tasks: speaker verification, voice style conversion, and a clinical application for detecting dysphonic voice. The experimental results demonstrate that adding an ICC regularizer can improve the repeatability of learned embeddings compared to only using the contrastive loss; further, these embeddings lead to improved performance in these downstream tasks.