# An Information-Theoretic Approach to Transferability in Task Transfer Learning

arXiv: https://arxiv.org/abs/2212.10082 · HTML: https://mlsanity.com/paper/2212.10082/

## Key facts

- Cited by 81 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 31 references that are in the map.
- First posted on arXiv 20 Dec 2022.

## Builds on (most cited first, 8 of 8)

- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [Deep Visual Domain Adaptation: A Survey](https://mlsanity.com/paper/1802.03601/) (2018)
- [Taskonomy: Disentangling Task Transfer Learning](https://mlsanity.com/paper/1804.08328/) (2018)
- [Supervised Learning of Universal Sentence Representations from Natural Language Inference Data](https://mlsanity.com/paper/1705.02364/) (2017)
- [DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition](https://mlsanity.com/paper/1310.1531/) (2013)
- [A PAC-Bayesian bound for Lifelong Learning](https://mlsanity.com/paper/1311.2838/) (2013)
- [An Efficient Approach to Informative Feature Extraction from Multimodal Data](https://mlsanity.com/paper/1811.08979/) (2018)
- [On Universal Features for High-Dimensional Learning and Inference](https://mlsanity.com/paper/1911.09105/) (2019)

## Built on by (most cited first, 30 of 81)

- [Sensitivity-Aware Visual Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2303.08566/) (2023)
- [Scaling Laws for Downstream Task Performance of Large Language Models](https://mlsanity.com/paper/2402.04177/) (2024)
- [Model Spider: Learning to Rank Pre-Trained Models Efficiently](https://mlsanity.com/paper/2306.03900/) (2023)
- [Capability Instruction Tuning: A New Paradigm for Dynamic LLM Routing](https://mlsanity.com/paper/2502.17282/) (2025)
- [Scalable Multitask Learning Using Gradient-based Estimation of Task Affinity](https://mlsanity.com/paper/2409.06091/) (2024)
- [Quick-Tune: Quickly Learning Which Pretrained Model to Finetune and How](https://mlsanity.com/paper/2306.03828/) (2023)
- [ETran: Energy-Based Transferability Estimation](https://mlsanity.com/paper/2308.02027/) (2023)
- [Which Model to Transfer? A Survey on Transferability Estimation](https://mlsanity.com/paper/2402.15231/) (2024)
- [Understanding the Transferability of Representations via Task-Relatedness](https://mlsanity.com/paper/2307.00823/) (2023)
- [Exploring Model Transferability through the Lens of Potential Energy](https://mlsanity.com/paper/2308.15074/) (2023)
- [Fast and Accurate Transferability Measurement by Evaluating Intra-class Feature Variance](https://mlsanity.com/paper/2308.05986/) (2023)
- [The Performance of Transferability Metrics does not Translate to Medical Tasks](https://mlsanity.com/paper/2308.07444/) (2023)
- [Model-Based Transfer Learning for Contextual Reinforcement Learning](https://mlsanity.com/paper/2408.04498/) (2024)
- [H-ensemble: An Information Theoretic Approach to Reliable Few-Shot Multi-Source-Free Transfer](https://mlsanity.com/paper/2312.12489/) (2023)
- [Building a Winning Team: Selecting Source Model Ensembles using a Submodular Transferability Estimation Approach](https://mlsanity.com/paper/2309.02429/) (2023)
- [ModelGiF: Gradient Fields for Model Functional Distance](https://mlsanity.com/paper/2309.11013/) (2023)
- [Simple Transferability Estimation for Regression Tasks](https://mlsanity.com/paper/2312.00656/) (2023)
- [A High-Dimensional Statistical Method for Optimizing Transfer Quantities in Multi-Source Transfer Learning](https://mlsanity.com/paper/2502.04242/) (2025)
- [Sequence Transferability and Task Order Selection in Continual Learning](https://mlsanity.com/paper/2502.06544/) (2025)
- [Distilling from Similar Tasks for Transfer Learning on a Budget](https://mlsanity.com/paper/2304.12314/) (2023)
- [How to Determine the Most Powerful Pre-trained Language Model without Brute Force Fine-tuning? An Empirical Survey](https://mlsanity.com/paper/2312.04775/) (2023)
- [Labeled Data Selection for Category Discovery](https://mlsanity.com/paper/2406.04898/) (2024)
- [BenTo: Benchmark Task Reduction with In-Context Transferability](https://mlsanity.com/paper/2410.13804/) (2024)
- [Understanding Knowledge Transferability for Transfer Learning: A Survey](https://mlsanity.com/paper/2507.03175/) (2025)
- [MELEP: A Novel Predictive Measure of Transferability in Multi-Label ECG Diagnosis](https://mlsanity.com/paper/2311.04224/) (2023)
- [LEAD: Exploring Logit Space Evolution for Model Selection](https://mlsanity.com/paper/2507.14559/) (2025)
- [One-Embedding-Fits-All: Efficient Zero-Shot Time Series Forecasting by a Model Zoo](https://mlsanity.com/paper/2509.04208/) (2025)
- [Representing LLMs in Prompt Semantic Task Space](https://mlsanity.com/paper/2509.22506/) (2025)
- [Transfer Learning for Benign Overfitting in High-Dimensional Linear Regression](https://mlsanity.com/paper/2510.15337/) (2025)
- [Feasibility and Transferability of Transfer Learning: A Mathematical Framework](https://mlsanity.com/paper/2301.11542/) (2023)

## Abstract (arXiv)

Task transfer learning is a popular technique in image processing applications that uses pre-trained models to reduce the supervision cost of related tasks. An important question is to determine task transferability, i.e. given a common input domain, estimating to what extent representations learned from a source task can help in learning a target task. Typically, transferability is either measured experimentally or inferred through task relatedness, which is often defined without a clear operational meaning. In this paper, we present a novel metric, H-score, an easily-computable evaluation function that estimates the performance of transferred representations from one task to another in classification problems using statistical and information theoretic principles. Experiments on real image data show that our metric is not only consistent with the empirical transferability measurement, but also useful to practitioners in applications such as source model selection and task transfer curriculum learning.