mlsanityData to 7 Oct 2026

MCRanker: Generating Diverse Criteria On-the-Fly to Improve Point-wise LLM Rankers

arXiv 2404.11960 ↗First posted 18 Apr 2024cs.IR cs.AIcited by 7

Not on the map. Why?

Abstract (arXiv)

The most recent pointwise Large Language Model (LLM) rankers have achieved remarkable ranking results. However, these rankers are hindered by two major drawbacks: (1) they fail to follow a standardized comparison guidance during the ranking process, and (2) they struggle with comprehensive considerations when dealing with complicated passages. To address these shortcomings, we propose to build a ranker that generates ranking scores based on a set of criteria from various perspectives. These criteria are intended to direct each perspective in providing a distinct yet synergistic evaluation. Our research, which examines eight datasets from the BEIR benchmark demonstrates that incorporating this multi-perspective criteria ensemble approach markedly enhanced the performance of pointwise LLM rankers.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 32 of its 40 references that are in the map. First posted on arXiv 18 Apr 2024.

Builds on

32 of 40 references.

References · 32

  1. 2020Document Ranking with a Pretrained Sequence-to-Sequence Modelcited by 373
  2. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 356
  3. 2023Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agentscited by 356
  4. 2023Large Language Models are Effective Text Rankers with Pairwise Ranking Promptingcited by 302
  5. 2023ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMscited by 297
  6. 2023Unleashing the Emergent Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaborationcited by 246
  7. 2023Exploring Collaboration Mechanisms for LLM Agents: A Social Psychology Viewcited by 221
  8. 2023Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedbackcited by 202
  9. 2019Multi-Stage Document Ranking with BERTcited by 200
  10. 2022Improving Passage Retrieval with Zero-Shot Question Generationcited by 164
  11. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  12. 2024Rethinking the Bounds of LLM Reasoning: Are Multi-Agent Discussions the Key?cited by 153
20 more
  1. 2023RankZephyr: Effective and Robust Zero-Shot Listwise Reranking is a Breeze!cited by 151
  2. 2024Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Modelscited by 150
  3. 2023ExpertPrompting: Instructing Large Language Models to be Distinguished Expertscited by 140
  4. 2023Large language models can accurately predict searcher preferencescited by 129
  5. 2024Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessmentcited by 117
  6. 2022RankT5: Fine-Tuning T5 for Text Ranking with Ranking Lossescited by 114
  7. 2023Beyond Yes and No: Improving Zero-Shot LLM Rankers via Scoring Fine-Grained Relevance Labelscited by 86
  8. 2023Perspectives on Large Language Models for Relevance Judgmentcited by 84
  9. 2023Zero-Shot Listwise Document Reranking with a Large Language Modelcited by 81
  10. 2023RankVicuna: Zero-Shot Listwise Document Reranking with Open-Source Large Language Modelscited by 69
  11. 2023A Setwise Approach for Effective and Highly Efficient Zero-shot Ranking with Large Language Modelscited by 53
  12. 2023Stance Detection with Collaborative Role-Infused LLM-Based Agentscited by 53
  13. 2023DERA: Enhancing Large Language Model Completions with Dialog-Enabled Resolving Agentscited by 52
  14. 2020PARADE: Passage Representation Aggregation for Document Rerankingcited by 44
  15. 2024LongAgent: Scaling Language Models to 128k Context through Multi-Agent Collaborationcited by 38
  16. 2020Learning-to-Rank with BERT in TF-Rankingcited by 36
  17. 2023Rank-without-GPT: Building GPT-Independent Listwise Rerankers on Open-Source Large Language Modelscited by 14
  18. 2020Best Practices for Managing Data Annotation Projectscited by 12
  19. 2023DORIS-MAE: Scientific Document Retrieval using Multi-level Aspect-based Queriescited by 11
  20. 2022Learning List-Level Domain-Invariant Representations for Rankingcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 3; 2026: 2.

7 papers.

  1. 2025Rank-R1: Enhancing Reasoning in LLM-based Document Rerankers via Reinforcement Learningcited by 58
  2. 2024TourRank: Utilizing Large Language Models for Documents Ranking with a Tournament-Inspired Strategycited by 47
  3. 2025A Survey of Multimodal Retrieval-Augmented Generationcited by 37
  4. 2024JudgeRank: Leveraging Large Language Models for Reasoning-Intensive Rerankingcited by 34
  5. 2025Likert or Not: LLM Absolute Relevance Judgments on Fine-Grained Ordinal Scalescited by 2
  6. 2026BracketRank: Large Language Model Document Ranking via Reasoning-based Competitive Eliminationcited by 0
  7. 2026ResRank: Unifying Retrieval and Listwise Reranking via End-to-End Joint Training with Residual Passage Compressioncited by 0