# LLM4Ranking: An Easy-to-use Framework of Utilizing Large Language Models for Document Reranking

arXiv: https://arxiv.org/abs/2504.07439 · HTML: https://mlsanity.com/paper/2504.07439/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 36 references that are in the map.
- First posted on arXiv 10 Apr 2025.

## Builds on (most cited first, 26 of 26)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [Passage Re-ranking with BERT](https://mlsanity.com/paper/1901.04085/) (2019)
- [8-bit Optimizers via Block-wise Quantization](https://mlsanity.com/paper/2110.02861/) (2021)
- [Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents](https://mlsanity.com/paper/2304.09542/) (2023)
- [Large Language Models for Information Retrieval: A Survey](https://mlsanity.com/paper/2308.07107/) (2023)
- [Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting](https://mlsanity.com/paper/2306.17563/) (2023)
- [Overview of the TREC 2019 deep learning track](https://mlsanity.com/paper/2003.07820/) (2020)
- [Document Expansion by Query Prediction](https://mlsanity.com/paper/1904.08375/) (2019)
- [BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval](https://mlsanity.com/paper/2407.12883/) (2024)
- [RankZephyr: Effective and Robust Zero-Shot Listwise Reranking is a Breeze!](https://mlsanity.com/paper/2312.02724/) (2023)
- [Beyond Yes and No: Improving Zero-Shot LLM Rankers via Scoring Fine-Grained Relevance Labels](https://mlsanity.com/paper/2310.14122/) (2023)
- [The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Models](https://mlsanity.com/paper/2101.05667/) (2021)
- [FIRST: Faster Improved Listwise Reranking with Single Token Decoding](https://mlsanity.com/paper/2406.15657/) (2024)
- [Tevatron: An Efficient and Flexible Toolkit for Dense Retrieval](https://mlsanity.com/paper/2203.05765/) (2022)
- [Questions Are All You Need to Train a Dense Passage Retriever](https://mlsanity.com/paper/2206.10658/) (2022)
- [TourRank: Utilizing Large Language Models for Documents Ranking with a Tournament-Inspired Strategy](https://mlsanity.com/paper/2406.11678/) (2024)
- [Attention in Large Language Models Yields Efficient Zero-Shot Re-Rankers](https://mlsanity.com/paper/2410.02642/) (2024)
- [NevIR: Negation in Neural Information Retrieval](https://mlsanity.com/paper/2305.07614/) (2023)
- [MAIR: A Massive Benchmark for Evaluating Instructed Retrieval](https://mlsanity.com/paper/2410.10127/) (2024)
- [DemoRank: Selecting Effective Demonstrations for Large Language Models in Ranking Task](https://mlsanity.com/paper/2406.16332/) (2024)
- [rerankers: A Lightweight Python Library to Unify Ranking Methods](https://mlsanity.com/paper/2408.17344/) (2024)

## Built on by (most cited first, 12 of 12)

- [mRAG: Elucidating the Design Space of Multi-modal Retrieval-Augmented Generation](https://mlsanity.com/paper/2505.24073/) (2025)
- [E2Rank: Unifying Text Embedding and Listwise Reranking for Effective and Efficient Search](https://mlsanity.com/paper/2510.22733/) (2025)
- [RLPO: Residual Listwise Preference Optimization for Long-Context Review Ranking](https://mlsanity.com/paper/2601.07449/) (2026)
- [DiffuRank: Effective Document Reranking with Diffusion Language Models](https://mlsanity.com/paper/2602.12528/) (2026)
- [SearchArt: Training Long-Horizon Search Agent with Scalable Synthetic and Verified Task](https://mlsanity.com/paper/2607.24850/) (2026)
- [Mitigating Hallucination in Large Language Models (LLMs): An Application-Oriented Survey on RAG, Reasoning, and Agentic Systems](https://mlsanity.com/paper/2510.24476/) (2025)
- [R1-Ranker: Teaching LLM Rankers to Reason](https://mlsanity.com/paper/2506.21638/) (2025)
- [RoutIR: Fast Serving of Retrieval Pipelines for Retrieval-Augmented Generation](https://mlsanity.com/paper/2601.10644/) (2026)
- [How Reliable are LLMs for Reasoning on the Re-ranking task?](https://mlsanity.com/paper/2508.18444/) (2025)
- [Scaling Search Relevance: Augmenting App Store Ranking with LLM-Generated Judgments](https://mlsanity.com/paper/2602.23234/) (2026)
- [DocRetriever: A Plug-and-Play Framework for Multimodal Document Retrieval with Comprehensive Benchmark](https://mlsanity.com/paper/2605.30027/) (2026)
- [PRISMR: Overcoming Parse Collapse in Multimodal Listwise Ranking via Parameterized Representation Internalization](https://mlsanity.com/paper/2606.12942/) (2026)

## Abstract (arXiv)

Utilizing large language models (LLMs) for document reranking has been a popular and promising research direction in recent years, many studies are dedicated to improving the performance and efficiency of using LLMs for reranking. Besides, it can also be applied in many real-world applications, such as search engines or retrieval-augmented generation. In response to the growing demand for research and application in practice, we introduce a unified framework, LLM4Ranking, which enables users to adopt different ranking methods using open-source or closed-source API-based LLMs. Our framework provides a simple and extensible interface for document reranking with LLMs, as well as easy-to-use evaluation and fine-tuning scripts for this task. We conducted experiments based on this framework and evaluated various models and methods on several widely used datasets, providing reproducibility results on utilizing LLMs for document reranking. Our code is publicly available at https://github.com/liuqi6777/llm4ranking.