# Few-shot Reranking for Multi-hop QA via Language Model Prompting

arXiv: https://arxiv.org/abs/2205.12650 · HTML: https://mlsanity.com/paper/2205.12650/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 54 references that are in the map.
- First posted on arXiv 25 May 2022.

## Builds on (most cited first, 30 of 40)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps](https://mlsanity.com/paper/2011.01060/) (2020)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [Reading Wikipedia to Answer Open-Domain Questions](https://mlsanity.com/paper/1704.00051/) (2017)
- [How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering](https://mlsanity.com/paper/2012.00955/) (2020)
- [Document Ranking with a Pretrained Sequence-to-Sequence Model](https://mlsanity.com/paper/2003.06713/) (2020)
- [Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration](https://mlsanity.com/paper/1910.12656/) (2019)
- [Calibration of Pre-trained Transformers](https://mlsanity.com/paper/2003.07892/) (2020)
- [Surface Form Competition: Why the Highest Probability Answer Isn't Always Right](https://mlsanity.com/paper/2104.08315/) (2021)
- [Distilling Knowledge from Reader to Retriever for Question Answering](https://mlsanity.com/paper/2012.04584/) (2020)
- [Improving Passage Retrieval with Zero-Shot Question Generation](https://mlsanity.com/paper/2204.07496/) (2022)
- [Answering Complex Open-Domain Questions with Multi-Hop Dense Retrieval](https://mlsanity.com/paper/2009.12756/) (2020)
- [Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering](https://mlsanity.com/paper/1911.10470/) (2019)
- [End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answering](https://mlsanity.com/paper/2106.05346/) (2021)
- [Unsupervised Question Decomposition for Question Answering](https://mlsanity.com/paper/2002.09758/) (2020)
- [RNNLogic: Learning Logic Rules for Reasoning on Knowledge Graphs](https://mlsanity.com/paper/2010.04029/) (2020)
- [Few-Shot Text Generation with Pattern-Exploiting Training](https://mlsanity.com/paper/2012.11926/) (2020)

## Built on by (most cited first, 19 of 19)

- [RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMs](https://mlsanity.com/paper/2407.02485/) (2024)
- [LongRAG: Enhancing Retrieval-Augmented Generation with Long-context LLMs](https://mlsanity.com/paper/2406.15319/) (2024)
- [PokeMQA: Programmable knowledge editing for Multi-hop Question Answering](https://mlsanity.com/paper/2312.15194/) (2023)
- [ChatGPT Rates Natural Language Explanation Quality Like Humans: But on Which Scales?](https://mlsanity.com/paper/2403.17368/) (2024)
- [Vision Search Assistant: Empower Vision-Language Models as Multimodal Search Engines](https://mlsanity.com/paper/2410.21220/) (2024)
- [Learning to Reason via Program Generation, Emulation, and Search](https://mlsanity.com/paper/2405.16337/) (2024)
- [ASRank: Zero-Shot Re-Ranking with Answer Scent for Document Retrieval](https://mlsanity.com/paper/2501.15245/) (2025)
- [SGIC: A Self-Guided Iterative Calibration Framework for RAG](https://mlsanity.com/paper/2506.16172/) (2025)
- [Few-shot Transfer Learning for Knowledge Base Question Answering: Fusing Supervised Models with In-Context Learning](https://mlsanity.com/paper/2311.08894/) (2023)
- [Automating construction contract review using knowledge graph-enhanced large language models](https://mlsanity.com/paper/2309.12132/) (2023)
- [Prompting Explicit and Implicit Knowledge for Multi-hop Question Answering Based on Human Reading Process](https://mlsanity.com/paper/2402.19350/) (2024)
- [Improving Retrieval Augmented Open-Domain Question-Answering with Vectorized Contexts](https://mlsanity.com/paper/2404.02022/) (2024)
- [Red Teaming Language Models for Processing Contradictory Dialogues](https://mlsanity.com/paper/2405.10128/) (2024)
- [GenSco: Can Question Decomposition based Passage Alignment improve Question Answering?](https://mlsanity.com/paper/2407.10245/) (2024)
- [LightPAL: Lightweight Passage Retrieval for Open Domain Multi-Document Summarization](https://mlsanity.com/paper/2406.12494/) (2024)
- [Learning to Extract Rational Evidence via Reinforcement Learning for Retrieval-Augmented Generation](https://mlsanity.com/paper/2507.15586/) (2025)
- [STRIDE: Strategic Iterative Decision-Making for Retrieval-Augmented Multi-Hop Question Answering](https://mlsanity.com/paper/2604.17405/) (2026)
- [DMA: Online RAG Alignment with Human Feedback](https://mlsanity.com/paper/2511.04880/) (2025)
- [Evaluating Prompt Engineering Techniques for RAG in Small Language Models: A Multi-Hop QA Approach](https://mlsanity.com/paper/2602.13890/) (2026)

## Abstract (arXiv)

We study few-shot reranking for multi-hop QA with open-domain questions. To alleviate the need for a large number of labeled question-document pairs for retriever training, we propose PromptRank, which relies on large language models prompting for multi-hop path reranking. PromptRank first constructs an instruction-based prompt that includes a candidate document path and then computes the relevance score between a given question and the path based on the conditional likelihood of the question given the path prompt according to a language model. PromptRank yields strong retrieval performance on HotpotQA with only 128 training examples compared to state-of-the-art methods trained on thousands of examples -- 73.6 recall@10 by PromptRank vs. 77.8 by PathRetriever and 77.5 by multi-hop dense retrieval. Code available at https://github.com/mukhal/PromptRank