# Duality Diagram Similarity: a generic framework for initialization selection in task transfer learning

arXiv: https://arxiv.org/abs/2008.02107 · HTML: https://mlsanity.com/paper/2008.02107/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Aug 2020.

## Built on by (most cited first, 23 of 23)

- [Scalable Diverse Model Selection for Accessible Transfer Learning](https://mlsanity.com/paper/2111.06977/) (2021)
- [Selecting Large Language Model to Fine-tune via Rectified Scaling Law](https://mlsanity.com/paper/2402.02314/) (2024)
- [Model Spider: Learning to Rank Pre-Trained Models Efficiently](https://mlsanity.com/paper/2306.03900/) (2023)
- [How stable are Transferability Metrics evaluations?](https://mlsanity.com/paper/2204.01403/) (2022)
- [Transferability Metrics for Selecting Source Model Ensembles](https://mlsanity.com/paper/2111.13011/) (2021)
- [Which Model to Transfer? A Survey on Transferability Estimation](https://mlsanity.com/paper/2402.15231/) (2024)
- [Understanding Robust Learning through the Lens of Representation Similarities](https://mlsanity.com/paper/2206.09868/) (2022)
- [Understanding the Transferability of Representations via Task-Relatedness](https://mlsanity.com/paper/2307.00823/) (2023)
- [Pick the Best Pre-trained Model: Towards Transferability Estimation for Medical Image Segmentation](https://mlsanity.com/paper/2307.11958/) (2023)
- [Simple Transferability Estimation for Regression Tasks](https://mlsanity.com/paper/2312.00656/) (2023)
- [Distilling from Similar Tasks for Transfer Learning on a Budget](https://mlsanity.com/paper/2304.12314/) (2023)
- [How to Determine the Most Powerful Pre-trained Language Model without Brute Force Fine-tuning? An Empirical Survey](https://mlsanity.com/paper/2312.04775/) (2023)
- [Understanding Knowledge Transferability for Transfer Learning: A Survey](https://mlsanity.com/paper/2507.03175/) (2025)
- [LEAD: Exploring Logit Space Evolution for Model Selection](https://mlsanity.com/paper/2507.14559/) (2025)
- [Exploring the Similarity of Representations in Model-Agnostic Meta-Learning](https://mlsanity.com/paper/2105.05757/) (2021)
- [CerberusDet: Unified Multi-Dataset Object Detection](https://mlsanity.com/paper/2407.12632/) (2024)
- [Benchmarking Transferability: A Framework for Fair and Robust Evaluation](https://mlsanity.com/paper/2504.20121/) (2025)
- [Pre-Trained Model Recommendation for Downstream Fine-tuning](https://mlsanity.com/paper/2403.06382/) (2024)
- [KITE: A Kernel-based Improved Transferability Estimation Method](https://mlsanity.com/paper/2405.01603/) (2024)
- [Similarity of Pre-trained and Fine-tuned Representations](https://mlsanity.com/paper/2207.09225/) (2022)
- [Back to the Basics on Predicting Transfer Performance](https://mlsanity.com/paper/2405.20420/) (2024)
- [Enabling Asymmetric Knowledge Transfer in Multi-Task Learning with Self-Auxiliaries](https://mlsanity.com/paper/2410.15875/) (2024)
- [Neural Coherence : Find higher performance to out-of-distribution tasks from few samples](https://mlsanity.com/paper/2512.05880/) (2025)

## Abstract (arXiv)

In this paper, we tackle an open research question in transfer learning, which is selecting a model initialization to achieve high performance on a new task, given several pre-trained models. We propose a new highly efficient and accurate approach based on duality diagram similarity (DDS) between deep neural networks (DNNs). DDS is a generic framework to represent and compare data of different feature dimensions. We validate our approach on the Taskonomy dataset by measuring the correspondence between actual transfer learning performance rankings on 17 taskonomy tasks and predicted rankings. Computing DDS based ranking for 17×17 transfers requires less than 2 minutes and shows a high correlation (0.86) with actual transfer learning rankings, outperforming state-of-the-art methods by a large margin (10%) on the Taskonomy benchmark. We also demonstrate the robustness of our model selection approach to a new task, namely Pascal VOC semantic segmentation. Additionally, we show that our method can be applied to select the best layer locations within a DNN for transfer learning on 2D, 3D and semantic tasks on NYUv2 and Pascal VOC datasets.