# MCRanker: Generating Diverse Criteria On-the-Fly to Improve Point-wise LLM Rankers

arXiv: https://arxiv.org/abs/2404.11960 · HTML: https://mlsanity.com/paper/2404.11960/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 40 references that are in the map.
- First posted on arXiv 18 Apr 2024.

## Builds on (most cited first, 30 of 32)

- [Document Ranking with a Pretrained Sequence-to-Sequence Model](https://mlsanity.com/paper/2003.06713/) (2020)
- [Toxicity in ChatGPT: Analyzing Persona-assigned Language Models](https://mlsanity.com/paper/2304.05335/) (2023)
- [Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents](https://mlsanity.com/paper/2304.09542/) (2023)
- [ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs](https://mlsanity.com/paper/2309.13007/) (2023)
- [Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting](https://mlsanity.com/paper/2306.17563/) (2023)
- [Unleashing the Emergent Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaboration](https://mlsanity.com/paper/2307.05300/) (2023)
- [Exploring Collaboration Mechanisms for LLM Agents: A Social Psychology View](https://mlsanity.com/paper/2310.02124/) (2023)
- [Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedback](https://mlsanity.com/paper/2305.10142/) (2023)
- [Multi-Stage Document Ranking with BERT](https://mlsanity.com/paper/1910.14424/) (2019)
- [Improving Passage Retrieval with Zero-Shot Question Generation](https://mlsanity.com/paper/2204.07496/) (2022)
- [Rethinking the Bounds of LLM Reasoning: Are Multi-Agent Discussions the Key?](https://mlsanity.com/paper/2402.18272/) (2024)
- [Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text](https://mlsanity.com/paper/2202.06935/) (2022)
- [RankZephyr: Effective and Robust Zero-Shot Listwise Reranking is a Breeze!](https://mlsanity.com/paper/2312.02724/) (2023)
- [Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Models](https://mlsanity.com/paper/2402.14207/) (2024)
- [ExpertPrompting: Instructing Large Language Models to be Distinguished Experts](https://mlsanity.com/paper/2305.14688/) (2023)
- [Large language models can accurately predict searcher preferences](https://mlsanity.com/paper/2309.10621/) (2023)
- [Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessment](https://mlsanity.com/paper/2402.14016/) (2024)
- [RankT5: Fine-Tuning T5 for Text Ranking with Ranking Losses](https://mlsanity.com/paper/2210.10634/) (2022)
- [Beyond Yes and No: Improving Zero-Shot LLM Rankers via Scoring Fine-Grained Relevance Labels](https://mlsanity.com/paper/2310.14122/) (2023)
- [Perspectives on Large Language Models for Relevance Judgment](https://mlsanity.com/paper/2304.09161/) (2023)
- [Zero-Shot Listwise Document Reranking with a Large Language Model](https://mlsanity.com/paper/2305.02156/) (2023)
- [RankVicuna: Zero-Shot Listwise Document Reranking with Open-Source Large Language Models](https://mlsanity.com/paper/2309.15088/) (2023)
- [A Setwise Approach for Effective and Highly Efficient Zero-shot Ranking with Large Language Models](https://mlsanity.com/paper/2310.09497/) (2023)
- [Stance Detection with Collaborative Role-Infused LLM-Based Agents](https://mlsanity.com/paper/2310.10467/) (2023)
- [DERA: Enhancing Large Language Model Completions with Dialog-Enabled Resolving Agents](https://mlsanity.com/paper/2303.17071/) (2023)
- [PARADE: Passage Representation Aggregation for Document Reranking](https://mlsanity.com/paper/2008.09093/) (2020)
- [LongAgent: Scaling Language Models to 128k Context through Multi-Agent Collaboration](https://mlsanity.com/paper/2402.11550/) (2024)
- [Learning-to-Rank with BERT in TF-Ranking](https://mlsanity.com/paper/2004.08476/) (2020)
- [Rank-without-GPT: Building GPT-Independent Listwise Rerankers on Open-Source Large Language Models](https://mlsanity.com/paper/2312.02969/) (2023)
- [Best Practices for Managing Data Annotation Projects](https://mlsanity.com/paper/2009.11654/) (2020)

## Built on by (most cited first, 7 of 7)

- [Rank-R1: Enhancing Reasoning in LLM-based Document Rerankers via Reinforcement Learning](https://mlsanity.com/paper/2503.06034/) (2025)
- [TourRank: Utilizing Large Language Models for Documents Ranking with a Tournament-Inspired Strategy](https://mlsanity.com/paper/2406.11678/) (2024)
- [A Survey of Multimodal Retrieval-Augmented Generation](https://mlsanity.com/paper/2504.08748/) (2025)
- [JudgeRank: Leveraging Large Language Models for Reasoning-Intensive Reranking](https://mlsanity.com/paper/2411.00142/) (2024)
- [Likert or Not: LLM Absolute Relevance Judgments on Fine-Grained Ordinal Scales](https://mlsanity.com/paper/2505.19334/) (2025)
- [BracketRank: Large Language Model Document Ranking via Reasoning-based Competitive Elimination](https://mlsanity.com/paper/2604.08834/) (2026)
- [ResRank: Unifying Retrieval and Listwise Reranking via End-to-End Joint Training with Residual Passage Compression](https://mlsanity.com/paper/2604.22180/) (2026)

## Abstract (arXiv)

The most recent pointwise Large Language Model (LLM) rankers have achieved remarkable ranking results. However, these rankers are hindered by two major drawbacks: (1) they fail to follow a standardized comparison guidance during the ranking process, and (2) they struggle with comprehensive considerations when dealing with complicated passages. To address these shortcomings, we propose to build a ranker that generates ranking scores based on a set of criteria from various perspectives. These criteria are intended to direct each perspective in providing a distinct yet synergistic evaluation. Our research, which examines eight datasets from the BEIR benchmark demonstrates that incorporating this multi-perspective criteria ensemble approach markedly enhanced the performance of pointwise LLM rankers.