mlsanityData to 14 Oct 2026

JudgeRank: Leveraging Large Language Models for Reasoning-Intensive Reranking

arXiv 2411.00142 ↗First posted 31 Oct 2024cs.CL cs.AIcited by 34

On the map: LLM reasoning → Knowledge- and tool-augmented reasoning

also in 1 more
Abstract (arXiv)

Accurate document retrieval is crucial for the success of retrieval-augmented generation (RAG) applications, including open-domain question answering and code completion. While large language models (LLMs) have been employed as dense encoders or listwise rerankers in RAG systems, they often struggle with reasoning-intensive tasks because they lack nuanced analysis when judging document relevance. To address this limitation, we introduce JudgeRank, a novel agentic reranker that emulates human cognitive processes when assessing document relevance. Our approach consists of three key steps: (1) query analysis to identify the core problem, (2) document analysis to extract a query-aware summary, and (3) relevance judgment to provide a concise assessment of document relevance. We evaluate JudgeRank on the reasoning-intensive BRIGHT benchmark, demonstrating substantial performance improvements over first-stage retrieval methods and outperforming other popular reranking approaches. In addition, JudgeRank performs on par with fine-tuned state-of-the-art rerankers on the popular BEIR benchmark, validating its zero-shot generalization capability. Through comprehensive ablation studies, we demonstrate that JudgeRank's performance generalizes well across LLMs of various sizes while ensembling them yields even more accurate reranking than individual models.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning. Builds on 33 of its 41 references that are in the map. First posted on arXiv 31 Oct 2024.

Builds on

33 of 41 references.

References · 33

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  7. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  8. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  9. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
  10. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,216
  11. 2019Passage Re-ranking with BERTcited by 645
  12. 2022MTEB: Massive Text Embedding Benchmarkcited by 640
21 more
  1. 2020Document Ranking with a Pretrained Sequence-to-Sequence Modelcited by 376
  2. 2023Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agentscited by 360
  3. 2023Large Language Models are Effective Text Rankers with Pairwise Ranking Promptingcited by 307
  4. 2023Fine-Tuning LLaMA for Multi-Stage Text Retrievalcited by 272
  5. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 268
  6. 2024RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMscited by 202
  7. 2024BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrievalcited by 158
  8. 2023RankZephyr: Effective and Robust Zero-Shot Listwise Reranking is a Breeze!cited by 155
  9. 2023Scaling Laws of RoPE-based Extrapolationcited by 125
  10. 2022RankT5: Fine-Tuning T5 for Text Ranking with Ranking Lossescited by 115
  11. 2023Zero-Shot Listwise Document Reranking with a Large Language Modelcited by 82
  12. 2021The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Modelscited by 81
  13. 2021Rethink Training of BERT Rerankers in Multi-Stage Retrieval Pipelinecited by 65
  14. 2023A Setwise Approach for Effective and Highly Efficient Zero-shot Ranking with Large Language Modelscited by 55
  15. 2023Found in the Middle: Permutation Self-Consistency Improves Listwise Ranking in Large Language Modelscited by 31
  16. 2024ListT5: Listwise Reranking with Fusion-in-Decoder Improves Zero-shot Retrievalcited by 30
  17. 2024Leveraging Passage Embeddings for Efficient Listwise Reranking with Large Language Modelscited by 20
  18. 2024A Thorough Comparison of Cross-Encoders and LLMs for Reranking SPLADEcited by 18
  19. 2023Resources for Brewing BEIR: Reproducible Reference Models and an Official Leaderboardcited by 15
  20. 2024rerankers: A Lightweight Python Library to Unify Ranking Methodscited by 8
  21. 2024MCRanker: Generating Diverse Criteria On-the-Fly to Improve Point-wise LLM Rankerscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 23; 2026: 10.

34 papers.

Sort:

By area:

  1. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  2. 2025Rank1: Test-Time Compute for Reranking in Information Retrievalcited by 70
  3. 2025Rank-R1: Enhancing Reasoning in LLM-based Document Rerankers via Reinforcement Learningcited by 60
  4. 2025ReasonEmbed: Enhanced Text Embeddings for Reasoning-Intensive Document Retrievalcited by 16
  5. 2025EnrichIndex: Using LLMs to Enrich Retrieval Indices Offlinecited by 15
  6. 2025Shifting from Ranking to Set Selection for Retrieval Augmented Generationcited by 15
  7. 2026Reasoning in Trees: Improving Retrieval-Augmented Generation for Multi-Hop Question Answeringcited by 14
  8. 2025MIR: Methodology Inspiration Retrieval for Scientific Research Problemscited by 11
  9. 2025TongSearch-QR: Reinforced Query Reasoning for Retrievalcited by 11
  10. 2025Optimizing Generative Ranking Relevance via Reinforcement Learning in Xiaohongshu Searchcited by 11
  11. 2025Large Reasoning Embedding Models: Towards Next-Generation Dense Retrieval Paradigmcited by 9
  12. 2025Judge as A Judge: Improving the Evaluation of Retrieval-Augmented Generation through the Judge-Consistency of Large Language Modelscited by 8
22 more
  1. 2025AcuRank: Uncertainty-Aware Adaptive Computation for Listwise Rerankingcited by 8
  2. 2025Retro*: Optimizing LLMs for Reasoning-Intensive Document Retrievalcited by 7
  3. 2025Rethinking Reasoning in Document Ranking: Why Chain-of-Thought Falls Shortcited by 7
  4. 2025Distillation and Refinement of Reasoning in Small Language Models for Document Re-rankingcited by 6
  5. 2025ERank: Fusing Supervised Fine-Tuning and Reinforcement Learning for Effective and Efficient Text Rerankingcited by 6
  6. 2026PersonalQ: Select, Quantize, and Serve Personalized Diffusion Models for Efficient Inferencecited by 6
  7. 2025TFRank: Think-Free Reasoning Enables Practical Pointwise LLM Rankingcited by 5
  8. 2025InsertRank: LLMs can reason over BM25 scores to Improve Listwise Rerankingcited by 4
  9. 2025E2Rank: Unifying Text Embedding and Listwise Reranking for Effective and Efficient Searchcited by 4
  10. 2025GroupRank: A Groupwise Paradigm for Effective and Efficient Passage Reranking with LLMscited by 3
  11. 2026NeocorRAG: Less Irrelevant Information, More Explicit Evidence, and More Effective Recall via Evidence Chainscited by 3
  12. 2025MAFA: A multi-agent framework for annotationcited by 2
  13. 2025RAR2: Retrieval-Augmented Medical Reasoning via Thought-Driven Retrievalcited by 2
  14. 2026Pailitao-VL: Unified Embedding and Reranker for Real-Time Multi-Modal Industrial Searchcited by 2
  15. 2025Hint-Augmented Re-ranking: Efficient Product Search using LLM-Based Query Decompositioncited by 1
  16. 2025Policy-based Sentence Simplification: Replacing Parallel Corpora with LLM-as-a-Judgecited by 1
  17. 2026A Survey of Reasoning-Intensive Retrieval: Progress and Challengescited by 1
  18. 2026Cross-Environment Neural Reranking for Sample-Efficient Action Selection in Text-Based Agentscited by 1
  19. 2026Understand and Accelerate Memory Processing Pipeline for Large Language Model Inferencecited by 0
  20. 2026IROH: Insightful Ranking Of Humor using Multi-Stage Hybrid Retrieval with Rationale-Distilled LLM Judges for JOKER 2026 Track Task 1 Englishcited by 0
  21. 2026Think Thrice Before Reranking: Multi-perspective Evidence and Reasoning Integration for Text Rerankingcited by 0
  22. 2026Component-Aware Feedback for Self-Evolving Programscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Prism-Reranker: Beyond Relevance Scoring -- Jointly Producing Contributions and Evidence for Agentic Retrieval
  2. 2025DynamicRAG: Leveraging Outputs of Large Language Model as Feedback for Dynamic Reranking in Retrieval-Augmented Generation
  3. 2025The Evolution of Reranking Models in Information Retrieval: From Heuristic Methods to Large Language Models
  4. 2026BracketRank: Large Language Model Document Ranking via Reasoning-based Competitive Elimination
  5. 2026Verbal-R3: Verbal Reranker as the Missing Bridge between Retrieval and Reasoning
  6. 2026LLM-Confidence Reranker: A Training-Free Approach for Enhancing Retrieval-Augmented Generation Systems
  7. 2025RankArena: A Unified Platform for Evaluating Retrieval, Reranking and RAG with Human and LLM Feedback
  8. 2026Tool-Adaptive LLM Reranker