# Representation Transfer by Optimal Transport

arXiv: https://arxiv.org/abs/2007.06737 · HTML: https://mlsanity.com/paper/2007.06737/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Jul 2020.

## Built on by (most cited first, 8 of 8)

- [Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks](https://mlsanity.com/paper/2207.13243/) (2022)
- [Fine-Tuning Graph Neural Networks via Graph Topology induced Optimal Transport](https://mlsanity.com/paper/2203.10453/) (2022)
- [Soft Matching Distance: A metric on neural representations that captures single-neuron tuning](https://mlsanity.com/paper/2311.09466/) (2023)
- [On Privileged and Convergent Bases in Neural Network Representations](https://mlsanity.com/paper/2307.12941/) (2023)
- [Differentiable Cluster Graph Neural Network](https://mlsanity.com/paper/2405.16185/) (2024)
- [A Lipschitz spaces view of infinitely wide shallow neural networks](https://mlsanity.com/paper/2410.14591/) (2024)
- [Connecting adversarial attacks and optimal transport for domain adaptation](https://mlsanity.com/paper/2205.15424/) (2022)
- [Beyond Structural Symmetries: Linear Mode Connectivity via Neuron Identifiability](https://mlsanity.com/paper/2606.04754/) (2026)

## Abstract (arXiv)

Learning generic representations with deep networks requires massive training samples and significant computer resources. To learn a new specific task, an important issue is to transfer the generic teacher's representation to a student network. In this paper, we propose to use a metric between representations that is based on a functional view of neurons. We use optimal transport to quantify the match between two representations, yielding a distance that embeds some invariances inherent to the representation of deep networks. This distance defines a regularizer promoting the similarity of the student's representation with that of the teacher. Our approach can be used in any learning context where representation transfer is applicable. We experiment here on two standard settings: inductive transfer learning, where the teacher's representation is transferred to a student network of same architecture for a new related task, and knowledge distillation, where the teacher's representation is transferred to a student of simpler architecture for the same task (model compression). Our approach also lends itself to solving new learning problems; we demonstrate this by showing how to directly transfer the teacher's representation to a simpler architecture student for a new related task.