# P^3 Ranker: Mitigating the Gaps between Pre-training and Ranking Fine-tuning with Prompt-based Learning and Pre-finetuning

arXiv: https://arxiv.org/abs/2205.01886 · HTML: https://mlsanity.com/paper/2205.01886/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 35 references that are in the map.
- First posted on arXiv 4 May 2022.

## Builds on (most cited first, 28 of 28)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [Reading Wikipedia to Answer Open-Domain Questions](https://mlsanity.com/paper/1704.00051/) (2017)
- [GPT Understands, Too](https://mlsanity.com/paper/2103.10385/) (2021)
- [Latent Retrieval for Weakly Supervised Open Domain Question Answering](https://mlsanity.com/paper/1906.00300/) (2019)
- [Passage Re-ranking with BERT](https://mlsanity.com/paper/1901.04085/) (2019)
- [Document Ranking with a Pretrained Sequence-to-Sequence Model](https://mlsanity.com/paper/2003.06713/) (2020)
- [AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts](https://mlsanity.com/paper/2010.15980/) (2020)
- [Multi-Stage Document Ranking with BERT](https://mlsanity.com/paper/1910.14424/) (2019)
- [Muppet: Massive Multi-task Representations with Pre-Finetuning](https://mlsanity.com/paper/2101.11038/) (2021)
- [Exploring and Predicting Transferability across NLP Tasks](https://mlsanity.com/paper/2005.00770/) (2020)
- [Intermediate-Task Transfer Learning with Pretrained Models for Natural Language Understanding: When and Why Does It Work?](https://mlsanity.com/paper/2005.00628/) (2020)
- [Deeper Text Understanding for IR with Contextual Neural Language Modeling](https://mlsanity.com/paper/1905.09217/) (2019)
- [Fine-grained Fact Verification with Kernel Graph Attention Network](https://mlsanity.com/paper/1910.09796/) (2019)
- [Open-Retrieval Conversational Question Answering](https://mlsanity.com/paper/2005.11364/) (2020)
- [End-to-End Neural Ad-hoc Ranking with Kernel Pooling](https://mlsanity.com/paper/1706.06613/) (2017)
- [MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehension](https://mlsanity.com/paper/1905.13453/) (2019)
- [Understanding the Behaviors of BERT in Ranking](https://mlsanity.com/paper/1904.07531/) (2019)
- [PROP: Pre-training with Representative Words Prediction for Ad-hoc Retrieval](https://mlsanity.com/paper/2010.10137/) (2020)
- [B-PROP: Bootstrapped Pre-training with Representative Words Prediction for Ad-hoc Retrieval](https://mlsanity.com/paper/2104.09791/) (2021)
- [Selective Weak Supervision for Neural Information Retrieval](https://mlsanity.com/paper/2001.10382/) (2020)
- [Few-Shot Text Ranking with Meta Adapted Synthetic Weak Supervision](https://mlsanity.com/paper/2012.14862/) (2020)

## Built on by (most cited first, 7 of 7)

- [Review of Large Vision Models and Visual Prompt Engineering](https://mlsanity.com/paper/2307.00855/) (2023)
- [GeoLayoutLM: Geometric Pre-training for Visual Information Extraction](https://mlsanity.com/paper/2304.10759/) (2023)
- [DUQGen: Effective Unsupervised Domain Adaptation of Neural Rankers by Diversifying Synthetic Query Generation](https://mlsanity.com/paper/2404.02489/) (2024)
- [Rethinking Dense Retrieval's Few-Shot Ability](https://mlsanity.com/paper/2304.05845/) (2023)
- [Annotating Data for Fine-Tuning a Neural Ranker? Current Active Learning Strategies are not Better than Random Selection](https://mlsanity.com/paper/2309.06131/) (2023)
- [Multi Self-supervised Pre-fine-tuned Transformer Fusion for Better Intelligent Transportation Detection](https://mlsanity.com/paper/2310.11307/) (2023)
- [Enhancing Document Information Analysis with Multi-Task Pre-training: A Robust Approach for Information Extraction in Visually-Rich Documents](https://mlsanity.com/paper/2310.16527/) (2023)

## Abstract (arXiv)

Compared to other language tasks, applying pre-trained language models (PLMs) for search ranking often requires more nuances and training signals. In this paper, we identify and study the two mismatches between pre-training and ranking fine-tuning: the training schema gap regarding the differences in training objectives and model architectures, and the task knowledge gap considering the discrepancy between the knowledge needed in ranking and that learned during pre-training. To mitigate these gaps, we propose Pre-trained, Prompt-learned and Pre-finetuned Neural Ranker (P^3 Ranker). P^3 Ranker leverages prompt-based learning to convert the ranking task into a pre-training like schema and uses pre-finetuning to initialize the model on intermediate supervised tasks. Experiments on MS MARCO and Robust04 show the superior performances of P^3 Ranker in few-shot ranking. Analyses reveal that P^3 Ranker is able to better accustom to the ranking task through prompt-based learning and retrieve necessary ranking-oriented knowledge gleaned in pre-finetuning, resulting in data-efficient PLM adaptation. Our code is available at https://github.com/NEUIR/P3Ranker.