mlsanityData to 7 Oct 2026

Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting

arXiv 2306.17563 ↗First posted 30 Jun 2023cs.IR cs.CL cs.LGcited by 302

Not on the map. Why?

Cited by 47 in Retrieval-augmented generation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Ranking documents using Large Language Models (LLMs) by directly feeding the query and candidate documents into the prompt is an interesting and practical problem. However, researchers have found it difficult to outperform fine-tuned baseline rankers on benchmark datasets. We analyze pointwise and listwise ranking prompts used by existing methods and argue that off-the-shelf LLMs do not fully understand these challenging ranking formulations. In this paper, we propose to significantly reduce the burden on LLMs by using a new technique called Pairwise Ranking Prompting (PRP). Our results are the first in the literature to achieve state-of-the-art ranking performance on standard benchmarks using moderate-sized open-sourced LLMs. On TREC-DL 2019&2020, PRP based on the Flan-UL2 model with 20B parameters performs favorably with the previous best approach in the literature, which is based on the blackbox commercial GPT-4 that has 50x (estimated) model size, while outperforming other LLM-based solutions, such as InstructGPT which has 175B parameters, by over 10% for all ranking metrics. By using the same prompt template on seven BEIR tasks, PRP outperforms supervised baselines and outperforms the blackbox commercial ChatGPT solution by 4.2% and pointwise LLM-based solutions by more than 10% on average NDCG@10. Furthermore, we propose several variants of PRP to improve efficiency and show that it is possible to achieve competitive results even with linear complexity.

Key facts. Cited by 302 papers in the map (data through 7 Oct 2026). Builds on 33 of its 55 references that are in the map. First posted on arXiv 30 Jun 2023.

Builds on

33 of 55 references.

References · 33

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  9. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  10. 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,122
  11. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  12. 2019Passage Re-ranking with BERTcited by 629
21 more
  1. 2022Precise Zero-Shot Dense Retrieval without Relevance Labelscited by 508
  2. 2020Document Ranking with a Pretrained Sequence-to-Sequence Modelcited by 373
  3. 2023Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agentscited by 356
  4. 2020Calibration of Pre-trained Transformerscited by 307
  5. 2023Large Language Models are Zero-Shot Rankers for Recommender Systemscited by 307
  6. 2022Transformer Memory as a Differentiable Search Indexcited by 242
  7. 2022Promptagator: Few-shot Dense Retrieval From 8 Examplescited by 233
  8. 2022Improving Passage Retrieval with Zero-Shot Question Generationcited by 164
  9. 2023Uncovering ChatGPT's Capabilities in Recommender Systemscited by 160
  10. 2023Query2doc: Query Expansion with Large Language Modelscited by 146
  11. 2022RankT5: Fine-Tuning T5 for Text Ranking with Ranking Lossescited by 114
  12. 2022A Neural Corpus Indexer for Document Retrievalcited by 107
  13. 2022Task-aware Retrieval with Instructionscited by 107
  14. 2023Zero-Shot Listwise Document Reranking with a Large Language Modelcited by 81
  15. 2015Simple, Robust and Optimal Ranking from Pairwise Comparisonscited by 68
  16. 2023Query Expansion by Prompting Large Language Modelscited by 56
  17. 2023One-Shot Labeling for Automatic Relevance Estimationcited by 31
  18. 2023Gpt-4: A Review on Advancements and Opportunities in Natural Language Processingcited by 15
  19. 2022ED2LM: Encoder-Decoder to Language Model for Faster Document Re-ranking Inferencecited by 9
  20. 2023RD-Suite: A Benchmark for Ranking Distillationcited by 7
  21. 2023ExaRanker: Explanation-Augmented Neural Rankercited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 27; 2024: 91; 2025: 110; 2026: 74.

302 papers.

Sort:

By area:

  1. 2023ChatDev: Communicative Agents for Software Developmentcited by 893
  2. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  3. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  4. 2023Large Language Models for Information Retrieval: A Surveycited by 331
  5. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  6. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
  7. 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 249
  8. 2024Scaling Large Language Model-based Multi-Agent Collaborationcited by 235
  9. 2024RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMscited by 200
  10. 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluatorscited by 146
  11. 2024Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessmentcited by 117
  12. 2024Bias and Unfairness in Information Retrieval Systems: New Challenges in the LLM Eracited by 97
38 more
  1. 2023MemoChat: Tuning LLMs to Use Memos for Consistent Long-Range Open-Domain Conversationcited by 94
  2. 2024LiPO: Listwise Preference Optimization through Learning-to-Rankcited by 94
  3. 2024CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklistscited by 87
  4. 2024A Survey on Retrieval-Augmented Text Generation for Large Language Modelscited by 77
  5. 2023RankVicuna: Zero-Shot Listwise Document Reranking with Open-Source Large Language Modelscited by 69
  6. 2024SuRe: Summarizing Retrievals using Answer Candidates for Open-domain QA of LLMscited by 67
  7. 2023LLM Comparative Assessment: Zero-shot NLG Evaluation through Pairwise Comparisons using Large Language Modelscited by 66
  8. 2025PathRAG: Pruning Graph-based Retrieval Augmented Generation with Relational Pathscited by 60
  9. 2024A Review of Modern Recommender Systems Using Generative Models (Gen-RecSys)cited by 58
  10. 2025Rank-R1: Enhancing Reasoning in LLM-based Document Rerankers via Reinforcement Learningcited by 58
  11. 2023Experiential Co-Learning of Software-Developing Agentscited by 57
  12. 2023A Setwise Approach for Effective and Highly Efficient Zero-shot Ranking with Large Language Modelscited by 53
  13. 2023LLatrieval: LLM-Verified Retrieval for Verifiable Generationcited by 52
  14. 2024Online Iterative Reinforcement Learning from Human Feedback with General Preference Modelcited by 52
  15. 2024The Limits of Inference Scaling Through Resamplingcited by 51
  16. 2024TourRank: Utilizing Large Language Models for Documents Ranking with a Tournament-Inspired Strategycited by 47
  17. 2024Improving Dynamic Object Interactions in Text-to-Video Generation with AI Feedbackcited by 45
  18. 2025Evaluating Scoring Bias in LLM-as-a-Judgecited by 42
  19. 2024The Larger the Better? Improved LLM Code-Generation via Budget Reallocationcited by 41
  20. 2025ReasonRank: Empowering Passage Ranking with Strong Reasoning Abilitycited by 39
  21. 2024CoRNStack: High-Quality Contrastive Data for Better Code Retrieval and Rerankingcited by 37
  22. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  23. 2025Improving LLM-as-a-Judge Inference with the Judgment Distributioncited by 36
  24. 2024JudgeRank: Leveraging Large Language Models for Reasoning-Intensive Rerankingcited by 34
  25. 2024Match, Compare, or Select? An Investigation of Large Language Models for Entity Matchingcited by 32
  26. 2025A Survey of Large Language Model Empowered Agents for Recommendation and Search: Towards Next-Generation Information Retrievalcited by 31
  27. 2023Found in the Middle: Permutation Self-Consistency Improves Listwise Ranking in Large Language Modelscited by 30
  28. 2024Prediction-Powered Ranking of Large Language Modelscited by 29
  29. 2024APEER: Automatic Prompt Engineering Enhances Large Language Model Rerankingcited by 28
  30. 2024Understand What LLM Needs: Dual Preference Alignment for Retrieval-Augmented Generationcited by 27
  31. 2024Fairer Preferences Elicit Improved Human-Aligned Large Language Model Judgmentscited by 25
  32. 2025REARANK: Reasoning Re-ranking Agent via Reinforcement Learningcited by 24
  33. 2025jina-reranker-v3: Last but Not Late Interaction for Listwise Document Rerankingcited by 23
  34. 2024Are Large Language Models Good at Utility Judgments?cited by 22
  35. 2024Efficient LLM Comparative Assessment: a Product of Experts Framework for Pairwise Comparisonscited by 22
  36. 2025Sentient Agent as a Judge: Evaluating Higher-Order Social Cognition in Large Language Modelscited by 21
  37. 2024Utilizing BERT for Information Retrieval: Survey, Applications, Resources, and Challengescited by 20
  38. 2024Ranked List Truncation for Large Language Model-based Re-Rankingcited by 20

50 most cited of 302 All 302 (JSON)