# Leveraging Estimated Transferability Over Human Intuition for Model Selection in Text Ranking

arXiv: https://arxiv.org/abs/2409.16198 · HTML: https://mlsanity.com/paper/2409.16198/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 33 of its 51 references that are in the map.
- First posted on arXiv 24 Sep 2024.

## Builds on (most cited first, 30 of 33)

- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval](https://mlsanity.com/paper/2007.00808/) (2020)
- [Fact or Fiction: Verifying Scientific Claims](https://mlsanity.com/paper/2004.14974/) (2020)
- [Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models](https://mlsanity.com/paper/2108.08877/) (2021)
- [The Power of Noise: Redefining Retrieval for RAG Systems](https://mlsanity.com/paper/2401.14887/) (2024)
- [Sparse, Dense, and Attentional Representations for Text Retrieval](https://mlsanity.com/paper/2005.00181/) (2020)
- [Whitening Sentence Representations for Better Semantics and Faster Retrieval](https://mlsanity.com/paper/2103.15316/) (2021)
- [LogME: Practical Assessment of Pre-trained Models for Transfer Learning](https://mlsanity.com/paper/2102.11005/) (2021)
- [MuTual: A Dataset for Multi-Turn Dialogue Reasoning](https://mlsanity.com/paper/2004.04494/) (2020)
- [From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective](https://mlsanity.com/paper/2205.04733/) (2022)
- [Ensembling Off-the-shelf Models for GAN Training](https://mlsanity.com/paper/2112.09130/) (2021)
- [An Information-Theoretic Approach to Transferability in Task Transfer Learning](https://mlsanity.com/paper/2212.10082/) (2022)
- [Relevance-guided Supervision for OpenQA with ColBERT](https://mlsanity.com/paper/2007.00814/) (2020)
- [WhiteningBERT: An Easy Unsupervised Sentence Embedding Approach](https://mlsanity.com/paper/2104.01767/) (2021)
- [Ranking Neural Checkpoints](https://mlsanity.com/paper/2011.11200/) (2020)
- [Scalable Transfer Learning with Expert Models](https://mlsanity.com/paper/2009.13239/) (2020)
- [An Empirical Study of Pre-Trained Model Reuse in the Hugging Face Deep Learning Model Registry](https://mlsanity.com/paper/2303.02552/) (2023)
- [Learning to Rank in Generative Retrieval](https://mlsanity.com/paper/2306.15222/) (2023)
- [Curriculum Learning for Dense Retrieval Distillation](https://mlsanity.com/paper/2204.13679/) (2022)
- [Evaluation of BERT and ALBERT Sentence Embedding Performance on Downstream NLP Tasks](https://mlsanity.com/paper/2101.10642/) (2021)
- [Model Spider: Learning to Rank Pre-Trained Models Efficiently](https://mlsanity.com/paper/2306.03900/) (2023)
- [PACTran: PAC-Bayesian Metrics for Estimating the Transferability of Pretrained Models to Classification Tasks](https://mlsanity.com/paper/2203.05126/) (2022)
- [How stable are Transferability Metrics evaluations?](https://mlsanity.com/paper/2204.01403/) (2022)
- [Not All Models Are Equal: Predicting Model Transferability in a Self-challenging Fisher Space](https://mlsanity.com/paper/2207.03036/) (2022)
- [PROD: Progressive Distillation for Dense Retrieval](https://mlsanity.com/paper/2209.13335/) (2022)
- [ConTextual Masked Auto-Encoder for Dense Passage Retrieval](https://mlsanity.com/paper/2208.07670/) (2022)
- [Foundation Model is Efficient Multimodal Multitask Model Selector](https://mlsanity.com/paper/2308.06262/) (2023)
- [DPTDR: Deep Prompt Tuning for Dense Passage Retrieval](https://mlsanity.com/paper/2208.11503/) (2022)

## Abstract (arXiv)

Text ranking has witnessed significant advancements, attributed to the utilization of dual-encoder enhanced by Pre-trained Language Models (PLMs). Given the proliferation of available PLMs, selecting the most effective one for a given dataset has become a non-trivial challenge. As a promising alternative to human intuition and brute-force fine-tuning, Transferability Estimation (TE) has emerged as an effective approach to model selection. However, current TE methods are primarily designed for classification tasks, and their estimated transferability may not align well with the objectives of text ranking. To address this challenge, we propose to compute the expected rank as transferability, explicitly reflecting the model's ranking capability. Furthermore, to mitigate anisotropy and incorporate training dynamics, we adaptively scale isotropic sentence embeddings to yield an accurate expected rank score. Our resulting method, Adaptive Ranking Transferability (AiRTran), can effectively capture subtle differences between models. On challenging model selection scenarios across various text ranking datasets, it demonstrates significant improvements over previous classification-oriented TE methods, human intuition, and ChatGPT with minor time consumption.