mlsanityData to 7 Oct 2026

Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents

arXiv 2304.09542 ↗First posted 19 Apr 2023cs.CL cs.IRcited by 356

On the map: LLM agents and tool use → Web, search, and research agents

Abstract (arXiv)

Large Language Models (LLMs) have demonstrated remarkable zero-shot generalization across various language-related tasks, including search engines. However, existing work utilizes the generative ability of LLMs for Information Retrieval (IR) rather than direct passage ranking. The discrepancy between the pre-training objectives of LLMs and the ranking objective poses another challenge. In this paper, we first investigate generative LLMs such as ChatGPT and GPT-4 for relevance ranking in IR. Surprisingly, our experiments reveal that properly instructed LLMs can deliver competitive, even superior results to state-of-the-art supervised methods on popular IR benchmarks. Furthermore, to address concerns about data contamination of LLMs, we collect a new test set called NovelEval, based on the latest knowledge and aiming to verify the model's ability to rank unknown knowledge. Finally, to improve efficiency in real-world applications, we delve into the potential for distilling the ranking capabilities of ChatGPT into small specialized models using a permutation distillation scheme. Our evaluation results turn out that a distilled 440M model outperforms a 3B supervised model on the BEIR benchmark. The code to reproduce our results is available at www.github.com/sunnweiwei/RankGPT.

Key facts. Cited by 356 papers in the map (data through 7 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 34 of its 46 references that are in the map. First posted on arXiv 19 Apr 2023.

Builds on

34 of 46 references.

References · 34

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  6. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  7. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  8. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  9. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  10. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  11. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
  12. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
22 more
  1. 2019Passage Re-ranking with BERTcited by 629
  2. 2022Precise Zero-Shot Dense Retrieval without Relevance Labelscited by 508
  3. 2020Document Ranking with a Pretrained Sequence-to-Sequence Modelcited by 373
  4. 2020Autoregressive Entity Retrievalcited by 333
  5. 2022Generate rather than Retrieve: Large Language Models are Strong Context Generatorscited by 296
  6. 2022Teaching Small Language Models to Reasoncited by 279
  7. 2020Pretrained Transformers for Text Ranking: BERT and Beyondcited by 266
  8. 2023Specializing Smaller Language Models towards Multi-Step Reasoningcited by 249
  9. 2022Transformer Memory as a Differentiable Search Indexcited by 242
  10. 2022Promptagator: Few-shot Dense Retrieval From 8 Examplescited by 233
  11. 2022SGPT: GPT Sentence Embeddings for Semantic Searchcited by 204
  12. 2021Overview of the TREC 2020 deep learning trackcited by 193
  13. 2022Improving Passage Retrieval with Zero-Shot Question Generationcited by 164
  14. 2021Mr. TyDi: A Multi-lingual Benchmark for Dense Retrievalcited by 118
  15. 2021mMARCO: A Multilingual Version of the MS MARCO Passage Ranking Datasetcited by 116
  16. 2022Task-aware Retrieval with Instructionscited by 107
  17. 2023Learning to Tokenize for Generative Retrievalcited by 66
  18. 2022InPars: Data Augmentation for Information Retrieval using Large Language Modelscited by 60
  19. 2022Questions Are All You Need to Train a Dense Passage Retrievercited by 47
  20. 2021Pre-trained Language Model based Ranking in Baidu Searchcited by 33
  21. 2021Shallow pooling for sparse labelscited by 13
  22. 2021Pre-training Methods in Information Retrievalcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 52; 2024: 115; 2025: 96; 2026: 93.

356 papers.

Sort:

By area:

  1. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  2. 2023A Survey on Large Language Models for Recommendationcited by 378
  3. 2023Large Language Models for Information Retrieval: A Surveycited by 331
  4. 2023Large Language Models are Effective Text Rankers with Pairwise Ranking Promptingcited by 302
  5. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  6. 2023The False Promise of Imitating Proprietary LLMscited by 254
  7. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Taskscited by 243
  8. 2024Generative Representational Instruction Tuningcited by 227
  9. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 224
  10. 2023MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Usecited by 209
  11. 2024RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMscited by 200
  12. 2024Fact-Checking the Output of Large Language Models via Token-Level Uncertainty Quantificationcited by 159
38 more
  1. 2024MM-Embed: Universal Multimodal Retrieval with Multimodal LLMscited by 159
  2. 2023Representation Learning with Large Language Models for Recommendationcited by 155
  3. 2024BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrievalcited by 153
  4. 2023Benchmarking Cognitive Biases in Large Language Models as Evaluatorscited by 147
  5. 2023How Can Recommender Systems Benefit from Large Language Models: A Surveycited by 141
  6. 2023A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasetscited by 123
  7. 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 123
  8. 2024DS-Agent: Automated Data Science by Empowering Large Language Models with Case-Based Reasoningcited by 119
  9. 2024HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughlycited by 119
  10. 2023MemoChat: Tuning LLMs to Use Memos for Consistent Long-Range Open-Domain Conversationcited by 94
  11. 2023ReLLa: Retrieval-enhanced Large Language Models for Lifelong Sequential Behavior Comprehension in Recommendationcited by 76
  12. 2025ReasonIR: Training Retrievers for Reasoning Taskscited by 76
  13. 2023GPTEval: A Survey on Assessments of ChatGPT and GPT-4cited by 73
  14. 2023Can Large Language Models Understand Real-World Complex Instructions?cited by 69
  15. 2023RankVicuna: Zero-Shot Listwise Document Reranking with Open-Source Large Language Modelscited by 69
  16. 2023Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarizationcited by 69
  17. 2024AvaTaR: Optimizing LLM Agents for Tool Usage via Contrastive Reasoningcited by 69
  18. 2023GPT4GEO: How a Language Model Sees the World's Geographycited by 68
  19. 2025Rank1: Test-Time Compute for Reranking in Information Retrievalcited by 68
  20. 2024SuRe: Summarizing Retrievals using Answer Candidates for Open-domain QA of LLMscited by 67
  21. 2024LitLLM: A Toolkit for Scientific Literature Reviewcited by 61
  22. 2024When Search Engine Services meet Large Language Models: Visions and Challengescited by 59
  23. 2025Rank-R1: Enhancing Reasoning in LLM-based Document Rerankers via Reinforcement Learningcited by 58
  24. 2026SkillRouter: Skill Routing for LLM Agents at Scalecited by 57
  25. 2023A Setwise Approach for Effective and Highly Efficient Zero-shot Ranking with Large Language Modelscited by 53
  26. 2023LlamaRec: Two-Stage Recommendation using Large Language Models for Rankingcited by 52
  27. 2023LLatrieval: LLM-Verified Retrieval for Verifiable Generationcited by 52
  28. 2024Generate-then-Ground in Retrieval-Augmented Generation for Multi-hop Question Answeringcited by 51
  29. 2024IDGenRec: LLM-RecSys Alignment with Textual ID Learningcited by 50
  30. 2024Don't Forget to Connect! Improving RAG with Graph-based Rerankingcited by 49
  31. 2024TourRank: Utilizing Large Language Models for Documents Ranking with a Tournament-Inspired Strategycited by 47
  32. 2024LitSearch: A Retrieval Benchmark for Scientific Literature Searchcited by 47
  33. 2024Attention in Large Language Models Yields Efficient Zero-Shot Re-Rankerscited by 42
  34. 2025Evaluating Scoring Bias in LLM-as-a-Judgecited by 42
  35. 2024The Larger the Better? Improved LLM Code-Generation via Budget Reallocationcited by 41
  36. 2026Skill Retrieval Augmentation for Agentic AIcited by 41
  37. 2023RETA-LLM: A Retrieval-Augmented Large Language Model Toolkitcited by 40
  38. 2024IM-RAG: Multi-Round Retrieval-Augmented Generation Through Learning Inner Monologuescited by 40

50 most cited of 356 All 356 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Language Ranker: A Lightweight Ranking framework for LLM Decoding
  2. 2026Aligning Large Language Models with Searcher Preferences
  3. 2026All Languages Matter: Understanding and Mitigating Language Bias in Multilingual RAG
  4. 2023Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models
  5. 2025From Retrieval to Generation: Comparing Different Approaches
  6. 2025RankLLM: A Python Package for Reranking with LLMs
  7. 2024Enhancing Q&A Text Retrieval with Ranking Models: Benchmarking, fine-tuning and deploying Rerankers for RAG
  8. 2025ProRank: Prompt Warmup via Reinforcement Learning for Small Language Models Reranking