# OTCE: A Transferability Metric for Cross-Domain Cross-Task Representations

arXiv: https://arxiv.org/abs/2103.13843 · HTML: https://mlsanity.com/paper/2103.13843/

## Key facts

- Cited by 62 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 44 references that are in the map.
- First posted on arXiv 25 Mar 2021.

## Builds on (most cited first, 21 of 21)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [Moment Matching for Multi-Source Domain Adaptation](https://mlsanity.com/paper/1812.01754/) (2018)
- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [Taskonomy: Disentangling Task Transfer Learning](https://mlsanity.com/paper/1804.08328/) (2018)
- [Cross-stitch Networks for Multi-task Learning](https://mlsanity.com/paper/1604.03539/) (2016)
- [Optimal Transport for Domain Adaptation](https://mlsanity.com/paper/1507.00504/) (2015)
- [DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition](https://mlsanity.com/paper/1310.1531/) (2013)
- [CNN Features off-the-shelf: an Astounding Baseline for Recognition](https://mlsanity.com/paper/1403.6382/) (2014)
- [Meta-Transfer Learning for Few-Shot Learning](https://mlsanity.com/paper/1812.02391/) (2018)
- [Task2Vec: Task Embedding for Meta-Learning](https://mlsanity.com/paper/1902.03545/) (2019)
- [UberNet: Training a `Universal' Convolutional Neural Network for Low-, Mid-, and High-Level Vision using Diverse Datasets and Limited Memory](https://mlsanity.com/paper/1609.02132/) (2016)
- [LEEP: A New Measure to Evaluate Transferability of Learned Representations](https://mlsanity.com/paper/2002.12462/) (2020)
- [Geometric Dataset Distances via Optimal Transport](https://mlsanity.com/paper/2002.02923/) (2020)
- [Transferability and Hardness of Supervised Classification Tasks](https://mlsanity.com/paper/1908.08142/) (2019)
- [HyperFace: A Deep Multi-task Learning Framework for Face Detection, Landmark Localization, Pose Estimation, and Gender Recognition](https://mlsanity.com/paper/1603.01249/) (2016)
- [Analyzing the Performance of Multilayer Neural Networks for Object Recognition](https://mlsanity.com/paper/1407.1610/) (2014)
- [Deep Cross Residual Learning for Multitask Visual Recognition](https://mlsanity.com/paper/1604.01335/) (2016)

## Built on by (most cited first, 30 of 62)

- [Recent Advances in Optimal Transport for Machine Learning](https://mlsanity.com/paper/2306.16156/) (2023)
- [Transferability Estimation using Bhattacharyya Class Separability](https://mlsanity.com/paper/2111.12780/) (2021)
- [Memory Efficient Continual Learning with Transformers](https://mlsanity.com/paper/2203.04640/) (2022)
- [Model Spider: Learning to Rank Pre-Trained Models Efficiently](https://mlsanity.com/paper/2306.03900/) (2023)
- [How stable are Transferability Metrics evaluations?](https://mlsanity.com/paper/2204.01403/) (2022)
- [Scalable Optimal Transport Methods in Machine Learning: A Contemporary Survey](https://mlsanity.com/paper/2305.05080/) (2023)
- [Capability Instruction Tuning: A New Paradigm for Dynamic LLM Routing](https://mlsanity.com/paper/2502.17282/) (2025)
- [Transferability Metrics for Selecting Source Model Ensembles](https://mlsanity.com/paper/2111.13011/) (2021)
- [How Well Do Supervised 3D Models Transfer to Medical Imaging Tasks?](https://mlsanity.com/paper/2501.11253/) (2025)
- [ETran: Energy-Based Transferability Estimation](https://mlsanity.com/paper/2308.02027/) (2023)
- [OT-VP: Optimal Transport-guided Visual Prompting for Test-Time Adaptation](https://mlsanity.com/paper/2407.09498/) (2024)
- [Newer is not always better: Rethinking transferability metrics, their peculiarities, stability and performance](https://mlsanity.com/paper/2110.06893/) (2021)
- [Transferability-Guided Cross-Domain Cross-Task Transfer Learning](https://mlsanity.com/paper/2207.05510/) (2022)
- [Deep transfer learning for image classification: a survey](https://mlsanity.com/paper/2205.09904/) (2022)
- [Which Model to Transfer? A Survey on Transferability Estimation](https://mlsanity.com/paper/2402.15231/) (2024)
- [A Bag-of-Prototypes Representation for Dataset-Level Applications](https://mlsanity.com/paper/2303.13251/) (2023)
- [Understanding the Transferability of Representations via Task-Relatedness](https://mlsanity.com/paper/2307.00823/) (2023)
- [When Neural Networks Fail to Generalize? A Model Sensitivity Perspective](https://mlsanity.com/paper/2212.00850/) (2022)
- [Lightspeed Geometric Dataset Distance via Sliced Optimal Transport](https://mlsanity.com/paper/2501.18901/) (2025)
- [Achieving Reliable and Fair Skin Lesion Diagnosis via Unsupervised Domain Adaptation](https://mlsanity.com/paper/2307.03157/) (2023)
- [Evidence > Intuition: Transferability Estimation for Encoder Selection](https://mlsanity.com/paper/2210.11255/) (2022)
- [Fair Classification via Domain Adaptation: A Dual Adversarial Learning Approach](https://mlsanity.com/paper/2206.03656/) (2022)
- [Building a Winning Team: Selecting Source Model Ensembles using a Submodular Transferability Estimation Approach](https://mlsanity.com/paper/2309.02429/) (2023)
- [Simple Transferability Estimation for Regression Tasks](https://mlsanity.com/paper/2312.00656/) (2023)
- [Sequence Transferability and Task Order Selection in Continual Learning](https://mlsanity.com/paper/2502.06544/) (2025)
- [Generalization Bounds for Deep Transfer Learning Using Majority Predictor Accuracy](https://mlsanity.com/paper/2209.05709/) (2022)
- [BenTo: Benchmark Task Reduction with In-Context Transferability](https://mlsanity.com/paper/2410.13804/) (2024)
- [Transferability Estimation for Semantic Segmentation Task](https://mlsanity.com/paper/2109.15242/) (2021)
- [Transferability Estimation Based On Principal Gradient Expectation](https://mlsanity.com/paper/2211.16299/) (2022)
- [One-Embedding-Fits-All: Efficient Zero-Shot Time Series Forecasting by a Model Zoo](https://mlsanity.com/paper/2509.04208/) (2025)

## Abstract (arXiv)

Transfer learning across heterogeneous data distributions (a.k.a. domains) and distinct tasks is a more general and challenging problem than conventional transfer learning, where either domains or tasks are assumed to be the same. While neural network based feature transfer is widely used in transfer learning applications, finding the optimal transfer strategy still requires time-consuming experiments and domain knowledge. We propose a transferability metric called Optimal Transport based Conditional Entropy (OTCE), to analytically predict the transfer performance for supervised classification tasks in such cross-domain and cross-task feature transfer settings. Our OTCE score characterizes transferability as a combination of domain difference and task difference, and explicitly evaluates them from data in a unified framework. Specifically, we use optimal transport to estimate domain difference and the optimal coupling between source and target distributions, which is then used to derive the conditional entropy of the target task (task difference). Experiments on the largest cross-domain dataset DomainNet and Office31 demonstrate that OTCE shows an average of 21% gain in the correlation with the ground truth transfer accuracy compared to state-of-the-art methods. We also investigate two applications of the OTCE score including source model selection and multi-source feature fusion.