# Learning Speaker Representations with Mutual Information

arXiv: https://arxiv.org/abs/1812.00271 · HTML: https://mlsanity.com/paper/1812.00271/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Dec 2018.

## Built on by (most cited first, 26 of 26)

- [BEATs: Audio Pre-Training with Acoustic Tokenizers](https://mlsanity.com/paper/2212.09058/) (2022)
- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [HDMI: High-order Deep Multiplex Infomax](https://mlsanity.com/paper/2102.07810/) (2021)
- [SLUE: New Benchmark Tasks for Spoken Language Understanding Evaluation on Natural Speech](https://mlsanity.com/paper/2111.10367/) (2021)
- [Audio Self-supervised Learning: A Survey](https://mlsanity.com/paper/2203.01205/) (2022)
- [VQMIVC: Vector Quantization and Mutual Information-Based Unsupervised Speech Representation Disentanglement for One-shot Voice Conversion](https://mlsanity.com/paper/2106.10132/) (2021)
- [Cross-modal Contrastive Learning for Speech Translation](https://mlsanity.com/paper/2205.02444/) (2022)
- [LiRA: Learning Visual Speech Representations from Audio through Self-supervision](https://mlsanity.com/paper/2106.09171/) (2021)
- [OPT: Omni-Perception Pre-Trainer for Cross-Modal Understanding and Generation](https://mlsanity.com/paper/2107.00249/) (2021)
- [Towards quantifying information flows: relative entropy in deep neural networks and the renormalization group](https://mlsanity.com/paper/2107.06898/) (2021)
- [AID-Purifier: A Light Auxiliary Network for Boosting Adversarial Defense](https://mlsanity.com/paper/2107.06456/) (2021)
- [Improving speech recognition models with small samples for air traffic control systems](https://mlsanity.com/paper/2102.08015/) (2021)
- [SKILL: Similarity-aware Knowledge distILLation for Speech Self-Supervised Learning](https://mlsanity.com/paper/2402.16830/) (2024)
- [Coherent, super resolved radar beamforming using self-supervised learning](https://mlsanity.com/paper/2106.13085/) (2021)
- [ProtoInfoMax: Prototypical Networks with Mutual Information Maximization for Out-of-Domain Detection](https://mlsanity.com/paper/2108.12229/) (2021)
- [Towards Supervised Performance on Speaker Verification with Self-Supervised Learning by Leveraging Large-Scale ASR Models](https://mlsanity.com/paper/2406.02285/) (2024)
- [Tune-In: Training Under Negative Environments with Interference for Attention Networks Simulating Cocktail Party Effect](https://mlsanity.com/paper/2103.01461/) (2021)
- [Learning Speaker Representation with Semi-supervised Learning approach for Speaker Profiling](https://mlsanity.com/paper/2110.13653/) (2021)
- [Speaker embeddings by modeling channel-wise correlations](https://mlsanity.com/paper/2104.02571/) (2021)
- [Multi network InfoMax: A pre-training method involving graph convolutional networks](https://mlsanity.com/paper/2111.01276/) (2021)
- [Bootstrap Equilibrium and Probabilistic Speaker Representation Learning for Self-supervised Speaker Verification](https://mlsanity.com/paper/2112.08929/) (2021)
- [NASTAR: Noise Adaptive Speech Enhancement with Target-Conditional Resampling](https://mlsanity.com/paper/2206.09058/) (2022)
- [Contrastive Learning from Synthetic Audio Doppelg"angers](https://mlsanity.com/paper/2406.05923/) (2024)
- [Toward Improving Synthetic Audio Spoofing Detection Robustness via Meta-Learning and Disentangled Training With Adversarial Examples](https://mlsanity.com/paper/2408.13341/) (2024)
- [Resounding Acoustic Fields with Reciprocity](https://mlsanity.com/paper/2510.20602/) (2025)
- [InfoShield: Privacy-Preserving Speech Representations for Mental Health Screening via Information-Theoretic Optimization](https://mlsanity.com/paper/2606.05561/) (2026)

## Abstract (arXiv)

Learning good representations is of crucial importance in deep learning. Mutual Information (MI) or similar measures of statistical dependence are promising tools for learning these representations in an unsupervised way. Even though the mutual information between two random variables is hard to measure directly in high dimensional spaces, some recent studies have shown that an implicit optimization of MI can be achieved with an encoder-discriminator architecture similar to that of Generative Adversarial Networks (GANs). In this work, we learn representations that capture speaker identities by maximizing the mutual information between the encoded representations of chunks of speech randomly sampled from the same sentence. The proposed encoder relies on the SincNet architecture and transforms raw speech waveform into a compact feature vector. The discriminator is fed by either positive samples (of the joint distribution of encoded chunks) or negative samples (from the product of the marginals) and is trained to separate them. We report experiments showing that this approach effectively learns useful speaker representations, leading to promising results on speaker identification and verification tasks. Our experiments consider both unsupervised and semi-supervised settings and compare the performance achieved with different objective functions.