mlsanityData to 7 Oct 2026

A Setwise Approach for Effective and Highly Efficient Zero-shot Ranking with Large Language Models

arXiv 2310.09497 ↗First posted 14 Oct 2023cs.IR cs.AIcited by 53

Not on the map. Why?

Abstract (arXiv)

We propose a novel zero-shot document ranking approach based on Large Language Models (LLMs): the Setwise prompting approach. Our approach complements existing prompting approaches for LLM-based zero-shot ranking: Pointwise, Pairwise, and Listwise. Through the first-of-its-kind comparative evaluation within a consistent experimental framework and considering factors like model size, token consumption, latency, among others, we show that existing approaches are inherently characterised by trade-offs between effectiveness and efficiency. We find that while Pointwise approaches score high on efficiency, they suffer from poor effectiveness. Conversely, Pairwise approaches demonstrate superior effectiveness but incur high computational overhead. Our Setwise approach, instead, reduces the number of LLM inferences and the amount of prompt token consumption during the ranking procedure, compared to previous methods. This significantly improves the efficiency of LLM-based zero-shot ranking, while also retaining high zero-shot ranking effectiveness. We make our code and results publicly available at urlhttps://github.com/ielab/llm-rankers.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). Builds on 25 of its 42 references that are in the map. First posted on arXiv 14 Oct 2023.

Builds on

25 of 42 references.

References · 25

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  9. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  10. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  11. 2023Large Language Models as Optimizerscited by 830
  12. 2023Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolutioncited by 521
13 more
  1. 2020Document Ranking with a Pretrained Sequence-to-Sequence Modelcited by 373
  2. 2023Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agentscited by 356
  3. 2023Large Language Models are Effective Text Rankers with Pairwise Ranking Promptingcited by 302
  4. 2020Overview of the TREC 2019 deep learning trackcited by 282
  5. 2020Pretrained Transformers for Text Ranking: BERT and Beyondcited by 266
  6. 2021Overview of the TREC 2020 deep learning trackcited by 193
  7. 2022Improving Passage Retrieval with Zero-Shot Question Generationcited by 164
  8. 2021The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Modelscited by 81
  9. 2023Zero-Shot Listwise Document Reranking with a Large Language Modelcited by 81
  10. 2023RankVicuna: Zero-Shot Listwise Document Reranking with Open-Source Large Language Modelscited by 69
  11. 2023Can ChatGPT Write a Good Boolean Query for Systematic Review Literature Search?cited by 54
  12. 2020Active Sampling for Pairwise Comparisons via Approximate Message Passing and Information Gain Maximizationcited by 17
  13. 2022Sparse Pairwise Re-ranking with Pre-trained Transformerscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 24; 2026: 21.

53 papers.

Sort:

By area:

  1. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,175
  2. 2024MM-Embed: Universal Multimodal Retrieval with Multimodal LLMscited by 159
  3. 2025Rank-R1: Enhancing Reasoning in LLM-based Document Rerankers via Reinforcement Learningcited by 58
  4. 2024Attention in Large Language Models Yields Efficient Zero-Shot Re-Rankerscited by 42
  5. 2024JudgeRank: Leveraging Large Language Models for Reasoning-Intensive Rerankingcited by 34
  6. 2026FinCARDS: Card-Based Analyst Reranking for Financial Document Question Answeringcited by 16
  7. 2025InfiGFusion: Graph-on-Logits Distillation via Efficient Gromov-Wasserstein for Model Fusioncited by 11
  8. 2025Set-LLM: A Permutation-Invariant LLMcited by 10
  9. 2025CLaRa: Bridging Retrieval and Generation with Continuous Latent Reasoningcited by 9
  10. 2025AcuRank: Uncertainty-Aware Adaptive Computation for Listwise Rerankingcited by 8
  11. 2024MCRanker: Generating Diverse Criteria On-the-Fly to Improve Point-wise LLM Rankerscited by 7
  12. 2024An Investigation of Prompt Variations for Zero-shot LLM-based Rankerscited by 7
38 more
  1. 2024Self-Calibrated Listwise Reranking with Large Language Modelscited by 7
  2. 2025Retro*: Optimizing LLMs for Reasoning-Intensive Document Retrievalcited by 7
  3. 2025Rank-Then-Score: Enhancing Large Language Models for Automated Essay Scoringcited by 6
  4. 2025Precise Zero-Shot Pointwise Ranking with LLMs through Post-Aggregated Global Context Informationcited by 6
  5. 2025Supervised Fine-Tuning or Contrastive Learning? Towards Better Multimodal LLM Rerankingcited by 6
  6. 2025A Multi-Task Evaluation of LLMs' Processing of Academic Text Inputcited by 5
  7. 2025Investigating Task Arithmetic for Zero-Shot Information Retrievalcited by 4
  8. 2025Fragile Preferences: A Deep Dive Into Order Effects in Large Language Modelscited by 4
  9. 2026One Pass, Any Order: Position-Invariant Listwise Reranking for LLM-Based Recommendationcited by 4
  10. 2025Beyond Reproducibility: Advancing Zero-shot LLM Reranking Efficiency with Setwise Insertioncited by 2
  11. 2025Likert or Not: LLM Absolute Relevance Judgments on Fine-Grained Ordinal Scalescited by 2
  12. 2026LANCER: LLM Reranking for Nugget Coveragecited by 2
  13. 2026Pailitao-VL: Unified Embedding and Reranker for Real-Time Multi-Modal Industrial Searchcited by 2
  14. 2026Beyond Relevance-Centric Retrieval: Rubric-Oriented Document Set Selection and Rankingcited by 2
  15. 2024Large Language Models for Stemming: Promises, Pitfalls and Failurescited by 1
  16. 2024GenSERP: Large Language Models for Whole Page Presentationcited by 1
  17. 2025JointRank: Rank Large Set with Single Passcited by 1
  18. 2025Access Paths for Efficient Ordering with Large Language Modelscited by 1
  19. 2025LLM Optimization Unlocks Real-Time Pairwise Rerankingcited by 1
  20. 2025Hint-Augmented Re-ranking: Efficient Product Search using LLM-Based Query Decompositioncited by 1
  21. 2025MixLM: High-Throughput and Effective LLM Ranking via Text-Embedding Mix-Interactioncited by 1
  22. 2026IRPM: Intergroup Relative Preference Modeling for Pointwise Generative Reward Modelscited by 1
  23. 2026An Efficient Long-Context Ranking Architecture With Calibrated LLM Distillation: Application to Person-Job Fitcited by 1
  24. 2026RoutIR: Fast Serving of Retrieval Pipelines for Retrieval-Augmented Generationcited by 1
  25. 2026LLM-Oriented Information Retrieval: A Denoising-First Perspectivecited by 1
  26. 2026ConFit v3: Improving Resume-Job Matching with LLM-based Re-Rankingcited by 1
  27. 2025Patent Figure Classification using Large Vision-language Modelscited by 0
  28. 2025MURR: Model Updating with Regularized Replay for Searching a Document Streamcited by 0
  29. 2025Predicting Movie Hits Before They Happen with LLMscited by 0
  30. 2025Jointly Generating and Attributing Answers using Logits of Document-Identifier Tokenscited by 0
  31. 2026BlitzRank: Principled Zero-shot Ranking Agents with Tournament Graphscited by 0
  32. 2026Scaling Search Relevance: Augmenting App Store Ranking with LLM-Generated Judgmentscited by 0
  33. 2026Task-Adaptive Embedding Refinement via Test-time LLM Guidancecited by 0
  34. 2026CompRank: Efficient LLM Reranking via Token-Level Compression and Decoding-Free Scoringcited by 0
  35. 2026Multimodal LLM-Empowered Re-Ranking for Generalizable Person Re-Identificationcited by 0
  36. 2026Shapley Context Pruning: A Cooperative Game Perspective for Context Reranking and Pruningcited by 0
  37. 2026When Do Anchor-Based Pointwise LLM Rerankers Help? Retriever Quality, Statistical Scope, and Anchor Designcited by 0
  38. 2026Action-Aligned Retrieval with Pairwise Multimodal Reranking for Text-Based Person Anomaly Searchcited by 0

50 most cited of 53 All 53 (JSON)