# Metric Learning with Progressive Self-Distillation for Audio-Visual Embedding Learning

arXiv: https://arxiv.org/abs/2501.09608 · HTML: https://mlsanity.com/paper/2501.09608/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation.
- Builds on 18 of its 33 references that are in the map.
- First posted on arXiv 16 Jan 2025.

## Placed in

- Knowledge distillation / [Speech and audio distillation](https://mlsanity.com/distill/speech-and-audio-distillation/)

## Builds on (most cited first, 18 of 18)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [In Defense of the Triplet Loss for Person Re-Identification](https://mlsanity.com/paper/1703.07737/) (2017)
- [Born Again Neural Networks](https://mlsanity.com/paper/1805.04770/) (2018)
- [Deep metric learning using Triplet network](https://mlsanity.com/paper/1412.6622/) (2014)
- [Visual to Sound: Generating Natural Sound for Videos in the Wild](https://mlsanity.com/paper/1712.01393/) (2017)
- [Self-Knowledge Distillation in Natural Language Processing](https://mlsanity.com/paper/1908.01851/) (2019)
- [Hardness-Aware Deep Metric Learning](https://mlsanity.com/paper/1903.05503/) (2019)
- [HyperML: A Boosting Metric Learning Approach in Hyperbolic Space for Recommender Systems](https://mlsanity.com/paper/1809.01703/) (2018)
- [It Takes Two to Tango: Mixup for Deep Metric Learning](https://mlsanity.com/paper/2106.04990/) (2021)
- [Deep Triplet Neural Networks with Cluster-CCA for Audio-Visual Cross-modal Retrieval](https://mlsanity.com/paper/1908.03737/) (2019)
- [Category-Based Deep CCA for Fine-Grained Venue Discovery from Multimodal Data](https://mlsanity.com/paper/1805.02997/) (2018)
- [Complete Cross-triplet Loss in Label Space for Audio-visual Cross-modal Retrieval](https://mlsanity.com/paper/2211.03434/) (2022)
- [VideoAdviser: Video Knowledge Distillation for Multimodal Transfer Learning](https://mlsanity.com/paper/2309.15494/) (2023)
- [Two-Stage Triplet Loss Training with Curriculum Augmentation for Audio-Visual Retrieval](https://mlsanity.com/paper/2310.13451/) (2023)
- [Anchor-aware Deep Metric Learning for Audio-visual Retrieval](https://mlsanity.com/paper/2404.13789/) (2024)
- [Learning Joint Embedding for Cross-Modal Retrieval](https://mlsanity.com/paper/1908.07673/) (2019)

## Built on by (most cited first, 3 of 3)

- [Learning Audio-Visual Embeddings with Inferred Latent Interaction Graphs](https://mlsanity.com/paper/2601.11995/) (2026)
- [Variance & Greediness: A comparative study of metric-learning losses](https://mlsanity.com/paper/2601.21450/) (2026)
- [Hierarchical Semantic Correlation-Aware Masked Autoencoder for Unsupervised Audio-Visual Representation Learning](https://mlsanity.com/paper/2604.04229/) (2026)

## Abstract (arXiv)

Metric learning projects samples into an embedded space, where similarities and dissimilarities are quantified based on their learned representations. However, existing methods often rely on label-guided representation learning, where representations of different modalities, such as audio and visual data, are aligned based on annotated labels. This approach tends to underutilize latent complex features and potential relationships inherent in the distributions of audio and visual data that are not directly tied to the labels, resulting in suboptimal performance in audio-visual embedding learning. To address this issue, we propose a novel architecture that integrates cross-modal triplet loss with progressive self-distillation. Our method enhances representation learning by leveraging inherent distributions and dynamically refining soft audio-visual alignments -- probabilistic alignments between audio and visual data that capture the inherent relationships beyond explicit labels. Specifically, the model distills audio-visual distribution-based knowledge from annotated labels in a subset of each batch. This self-distilled knowledge is used t