# Contrastive Fine-tuning Improves Robustness for Neural Rankers

arXiv: https://arxiv.org/abs/2105.12932 · HTML: https://mlsanity.com/paper/2105.12932/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 54 references that are in the map.
- First posted on arXiv 27 May 2021.

## Builds on (most cited first, 30 of 39)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval](https://mlsanity.com/paper/2007.00808/) (2020)
- [EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks](https://mlsanity.com/paper/1901.11196/) (2019)
- [In Defense of the Triplet Loss for Person Re-Identification](https://mlsanity.com/paper/1703.07737/) (2017)
- [Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment](https://mlsanity.com/paper/1907.11932/) (2019)
- [Using Self-Supervised Learning Can Improve Model Robustness and Uncertainty](https://mlsanity.com/paper/1906.12340/) (2019)
- [Learning the Difference that Makes a Difference with Counterfactually-Augmented Data](https://mlsanity.com/paper/1909.12434/) (2019)
- [Keystroke Dynamics as Part of Lifelogging](https://mlsanity.com/paper/2010.10144/) (2020)
- [Bridging Theory and Algorithm for Domain Adaptation](https://mlsanity.com/paper/1904.05801/) (2019)
- [CLEAR: Contrastive Learning for Sentence Representation](https://mlsanity.com/paper/2012.15466/) (2020)
- [Learning Dense Representations for Entity Retrieval](https://mlsanity.com/paper/1909.10506/) (2019)
- [Deep Metric Learning with Angular Loss](https://mlsanity.com/paper/1708.01682/) (2017)
- [Robust Neural Machine Translation with Doubly Adversarial Inputs](https://mlsanity.com/paper/1906.02443/) (2019)
- [Cycle-Consistency for Robust Visual Question Answering](https://mlsanity.com/paper/1902.05660/) (2019)
- [Multi-passage BERT: A Globally Normalized BERT Model for Open-domain Question Answering](https://mlsanity.com/paper/1908.08167/) (2019)
- [Syntactic Data Augmentation Increases Robustness to Inference Heuristics](https://mlsanity.com/paper/2004.11999/) (2020)
- [Adv-BERT: BERT is not robust on misspellings! Generating nature adversarial samples on BERT](https://mlsanity.com/paper/2003.04985/) (2020)
- [PyTorch Metric Learning](https://mlsanity.com/paper/2008.09164/) (2020)
- [Robust Machine Comprehension Models via Adversarial Training](https://mlsanity.com/paper/1804.06473/) (2018)
- [Attentive Pooling Networks](https://mlsanity.com/paper/1602.03609/) (2016)
- [Applying Deep Learning to Answer Selection: A Study and An Open Task](https://mlsanity.com/paper/1508.01585/) (2015)
- [Beyond [CLS] through Ranking by Generation](https://mlsanity.com/paper/2010.03073/) (2020)

## Built on by (most cited first, 12 of 12)

- [Successive Prompting for Decomposing Complex Questions](https://mlsanity.com/paper/2212.04092/) (2022)
- [Heterogeneous Contrastive Learning for Foundation Models and Beyond](https://mlsanity.com/paper/2404.00225/) (2024)
- [Robust Neural Information Retrieval: An Adversarial and Out-of-distribution Perspective](https://mlsanity.com/paper/2407.06992/) (2024)
- [Improving Multimodal Sentiment Analysis: Supervised Angular Margin-based Contrastive Learning for Enhanced Fusion Representation](https://mlsanity.com/paper/2312.02227/) (2023)
- [A Study on FGSM Adversarial Training for Neural Retrieval](https://mlsanity.com/paper/2301.10576/) (2023)
- [MAMA: Meta-optimized Angular Margin Contrastive Framework for Video-Language Representation Learning](https://mlsanity.com/paper/2407.03788/) (2024)
- [Continual Learning of Long Topic Sequences in Neural Information Retrieval](https://mlsanity.com/paper/2201.03356/) (2022)
- [DialAug: Mixing up Dialogue Contexts in Contrastive Learning for Robust Conversational Modeling](https://mlsanity.com/paper/2204.07679/) (2022)
- [Defense of Adversarial Ranking Attack in Text Retrieval: Benchmark and Baseline via Detection](https://mlsanity.com/paper/2307.16816/) (2023)
- [Exploring Representation Invariance in Finetuning](https://mlsanity.com/paper/2503.07399/) (2025)
- [QBR: A Question-Bank-Based Approach to Fine-Grained Legal Knowledge Retrieval for the General Public](https://mlsanity.com/paper/2505.04883/) (2025)
- [Towards Motion-aware Referring Image Segmentation](https://mlsanity.com/paper/2603.17413/) (2026)

## Abstract (arXiv)

The performance of state-of-the-art neural rankers can deteriorate substantially when exposed to noisy inputs or applied to a new domain. In this paper, we present a novel method for fine-tuning neural rankers that can significantly improve their robustness to out-of-domain data and query perturbations. Specifically, a contrastive loss that compares data points in the representation space is combined with the standard ranking loss during fine-tuning. We use relevance labels to denote similar/dissimilar pairs, which allows the model to learn the underlying matching semantics across different query-document pairs and leads to improved robustness. In experiments with four passage ranking datasets, the proposed contrastive fine-tuning method obtains improvements on robustness to query reformulations, noise perturbations, and zero-shot transfer for both BERT and BART based rankers. Additionally, our experiments show that contrastive fine-tuning outperforms data augmentation for robustifying neural rankers.