# A Setwise Approach for Effective and Highly Efficient Zero-shot Ranking with Large Language Models

arXiv: https://arxiv.org/abs/2310.09497 · HTML: https://mlsanity.com/paper/2310.09497/

## Key facts

- Cited by 55 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 42 references that are in the map.
- First posted on arXiv 14 Oct 2023.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://mlsanity.com/paper/2212.03533/) (2022)
- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [Large Language Models as Optimizers](https://mlsanity.com/paper/2309.03409/) (2023)
- [Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution](https://mlsanity.com/paper/2309.16797/) (2023)
- [Document Ranking with a Pretrained Sequence-to-Sequence Model](https://mlsanity.com/paper/2003.06713/) (2020)
- [Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents](https://mlsanity.com/paper/2304.09542/) (2023)
- [Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting](https://mlsanity.com/paper/2306.17563/) (2023)
- [Overview of the TREC 2019 deep learning track](https://mlsanity.com/paper/2003.07820/) (2020)
- [Pretrained Transformers for Text Ranking: BERT and Beyond](https://mlsanity.com/paper/2010.06467/) (2020)
- [Overview of the TREC 2020 deep learning track](https://mlsanity.com/paper/2102.07662/) (2021)
- [Improving Passage Retrieval with Zero-Shot Question Generation](https://mlsanity.com/paper/2204.07496/) (2022)
- [Zero-Shot Listwise Document Reranking with a Large Language Model](https://mlsanity.com/paper/2305.02156/) (2023)
- [The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Models](https://mlsanity.com/paper/2101.05667/) (2021)
- [RankVicuna: Zero-Shot Listwise Document Reranking with Open-Source Large Language Models](https://mlsanity.com/paper/2309.15088/) (2023)
- [Can ChatGPT Write a Good Boolean Query for Systematic Review Literature Search?](https://mlsanity.com/paper/2302.03495/) (2023)
- [Active Sampling for Pairwise Comparisons via Approximate Message Passing and Information Gain Maximization](https://mlsanity.com/paper/2004.05691/) (2020)
- [Sparse Pairwise Re-ranking with Pre-trained Transformers](https://mlsanity.com/paper/2207.04470/) (2022)

## Built on by (most cited first, 30 of 55)

- [Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models](https://mlsanity.com/paper/2506.05176/) (2025)
- [MM-Embed: Universal Multimodal Retrieval with Multimodal LLMs](https://mlsanity.com/paper/2411.02571/) (2024)
- [Rank-R1: Enhancing Reasoning in LLM-based Document Rerankers via Reinforcement Learning](https://mlsanity.com/paper/2503.06034/) (2025)
- [Attention in Large Language Models Yields Efficient Zero-Shot Re-Rankers](https://mlsanity.com/paper/2410.02642/) (2024)
- [JudgeRank: Leveraging Large Language Models for Reasoning-Intensive Reranking](https://mlsanity.com/paper/2411.00142/) (2024)
- [FinCARDS: Card-Based Analyst Reranking for Financial Document Question Answering](https://mlsanity.com/paper/2601.06992/) (2026)
- [InfiGFusion: Graph-on-Logits Distillation via Efficient Gromov-Wasserstein for Model Fusion](https://mlsanity.com/paper/2505.13893/) (2025)
- [Set-LLM: A Permutation-Invariant LLM](https://mlsanity.com/paper/2505.15433/) (2025)
- [CLaRa: Bridging Retrieval and Generation with Continuous Latent Reasoning](https://mlsanity.com/paper/2511.18659/) (2025)
- [An Investigation of Prompt Variations for Zero-shot LLM-based Rankers](https://mlsanity.com/paper/2406.14117/) (2024)
- [AcuRank: Uncertainty-Aware Adaptive Computation for Listwise Reranking](https://mlsanity.com/paper/2505.18512/) (2025)
- [MCRanker: Generating Diverse Criteria On-the-Fly to Improve Point-wise LLM Rankers](https://mlsanity.com/paper/2404.11960/) (2024)
- [Self-Calibrated Listwise Reranking with Large Language Models](https://mlsanity.com/paper/2411.04602/) (2024)
- [Retro*: Optimizing LLMs for Reasoning-Intensive Document Retrieval](https://mlsanity.com/paper/2509.24869/) (2025)
- [Rank-Then-Score: Enhancing Large Language Models for Automated Essay Scoring](https://mlsanity.com/paper/2504.05736/) (2025)
- [Precise Zero-Shot Pointwise Ranking with LLMs through Post-Aggregated Global Context Information](https://mlsanity.com/paper/2506.10859/) (2025)
- [Supervised Fine-Tuning or Contrastive Learning? Towards Better Multimodal LLM Reranking](https://mlsanity.com/paper/2510.14824/) (2025)
- [Fragile Preferences: A Deep Dive Into Order Effects in Large Language Models](https://mlsanity.com/paper/2506.14092/) (2025)
- [A Multi-Task Evaluation of LLMs' Processing of Academic Text Input](https://mlsanity.com/paper/2508.11779/) (2025)
- [One Pass, Any Order: Position-Invariant Listwise Reranking for LLM-Based Recommendation](https://mlsanity.com/paper/2604.27599/) (2026)
- [Investigating Task Arithmetic for Zero-Shot Information Retrieval](https://mlsanity.com/paper/2505.00649/) (2025)
- [Likert or Not: LLM Absolute Relevance Judgments on Fine-Grained Ordinal Scales](https://mlsanity.com/paper/2505.19334/) (2025)
- [Beyond Relevance-Centric Retrieval: Rubric-Oriented Document Set Selection and Ranking](https://mlsanity.com/paper/2607.19747/) (2026)
- [Beyond Reproducibility: Advancing Zero-shot LLM Reranking Efficiency with Setwise Insertion](https://mlsanity.com/paper/2504.10509/) (2025)
- [LANCER: LLM Reranking for Nugget Coverage](https://mlsanity.com/paper/2601.22008/) (2026)
- [Pailitao-VL: Unified Embedding and Reranker for Real-Time Multi-Modal Industrial Search](https://mlsanity.com/paper/2602.13704/) (2026)
- [ConFit v3: Improving Resume-Job Matching with LLM-based Re-Ranking](https://mlsanity.com/paper/2605.09760/) (2026)
- [Large Language Models for Stemming: Promises, Pitfalls and Failures](https://mlsanity.com/paper/2402.11757/) (2024)
- [GenSERP: Large Language Models for Whole Page Presentation](https://mlsanity.com/paper/2402.14301/) (2024)
- [JointRank: Rank Large Set with Single Pass](https://mlsanity.com/paper/2506.22262/) (2025)

## Abstract (arXiv)

We propose a novel zero-shot document ranking approach based on Large Language Models (LLMs): the Setwise prompting approach. Our approach complements existing prompting approaches for LLM-based zero-shot ranking: Pointwise, Pairwise, and Listwise. Through the first-of-its-kind comparative evaluation within a consistent experimental framework and considering factors like model size, token consumption, latency, among others, we show that existing approaches are inherently characterised by trade-offs between effectiveness and efficiency. We find that while Pointwise approaches score high on efficiency, they suffer from poor effectiveness. Conversely, Pairwise approaches demonstrate superior effectiveness but incur high computational overhead. Our Setwise approach, instead, reduces the number of LLM inferences and the amount of prompt token consumption during the ranking procedure, compared to previous methods. This significantly improves the efficiency of LLM-based zero-shot ranking, while also retaining high zero-shot ranking effectiveness. We make our code and results publicly available at urlhttps://github.com/ielab/llm-rankers.