mlsanityData to 14 Oct 2026

Rankers, Judges, and Assistants: Towards Understanding the Interplay of LLMs in Information Retrieval Evaluation

arXiv 2503.19092 ↗First posted 24 Mar 2025cs.IR cs.AI cs.CLcited by 15

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) are increasingly integral to information retrieval (IR), powering ranking, evaluation, and AI-assisted content creation. This widespread adoption necessitates a critical examination of potential biases arising from the interplay between these LLM-based components. This paper synthesizes existing research and presents novel experiment designs that explore how LLM-based rankers and assistants influence LLM-based judges. We provide the first empirical evidence of LLM judges exhibiting significant bias towards LLM-based rankers. Furthermore, we observe limitations in LLM judges' ability to discern subtle system performance differences. Contrary to some previous findings, our preliminary study does not find evidence of bias against AI-generated content. These results highlight the need for a more holistic view of the LLM-driven information ecosystem. To this end, we offer initial guidelines and a research agenda to ensure the reliable use of LLMs in IR evaluation.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 54 of its 72 references that are in the map. First posted on arXiv 24 Mar 2025.

Builds on

54 of 72 references.

References · 54

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  4. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  5. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  6. 2024A Survey on LLM-as-a-Judgecited by 1,423
  7. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
  8. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
  9. 2019Passage Re-ranking with BERTcited by 645
  10. 2023Bias and Fairness in Large Language Models: A Surveycited by 639
  11. 2019Text Summarization with Pretrained Encoderscited by 556
  12. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 418
38 more
  1. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 408
  2. 2020Document Ranking with a Pretrained Sequence-to-Sequence Modelcited by 376
  3. 2023Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agentscited by 360
  4. 2023Large Language Models are Effective Text Rankers with Pairwise Ranking Promptingcited by 307
  5. 2021Towards Understanding and Mitigating Social Biases in Language Modelscited by 296
  6. 2020Overview of the TREC 2019 deep learning trackcited by 286
  7. 2022Transformer Memory as a Differentiable Search Indexcited by 247
  8. 2022Promptagator: Few-shot Dense Retrieval From 8 Examplescited by 235
  9. 2021Overview of the TREC 2020 deep learning trackcited by 196
  10. 2022Improving Passage Retrieval with Zero-Shot Question Generationcited by 165
  11. 2024Large Language Models are Inconsistent and Biased Evaluatorscited by 153
  12. 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluatorscited by 149
  13. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHFcited by 136
  14. 2023Large language models can accurately predict searcher preferencescited by 131
  15. 2022RankT5: Fine-Tuning T5 for Text Ranking with Ranking Lossescited by 115
  16. 2024A Note on Shumailov et al. (2024): `AI Models Collapse When Trained on Recursively Generated Data'cited by 115
  17. 2024Pride and Prejudice: LLM Amplifies Self-Bias in Self-Refinementcited by 114
  18. 2024Bias and Unfairness in Information Retrieval Systems: New Challenges in the LLM Eracited by 98
  19. 2023LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scorescited by 90
  20. 2023Perspectives on Large Language Models for Relevance Judgmentcited by 85
  21. 2021The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Modelscited by 81
  22. 2024Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluationcited by 74
  23. 2022InPars: Data Augmentation for Information Retrieval using Large Language Modelscited by 62
  24. 2025The FACTS Grounding Leaderboard: Benchmarking LLMs' Ability to Ground Responses to Long-Form Inputcited by 52
  25. 2025The Widespread Adoption of Large Language Model-Assisted Writing Across Societycited by 51
  26. 2024UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessorcited by 42
  27. 2023Neural Retrievers are Biased Towards LLM-Generated Contentcited by 38
  28. 2023One-Shot Labeling for Automatic Relevance Estimationcited by 31
  29. 2024LLM-based relevance assessment still can't replace human relevance assessmentcited by 27
  30. 2024Don't Use LLMs to Make Relevance Judgmentscited by 26
  31. 2024A Survey on Recent Advances in Conversational Data Generationcited by 23
  32. 2024AI Can Be Cognitively Biased: An Exploratory Study on Threshold Priming in LLM-Based Batch Relevance Assessmentcited by 20
  33. 2024A Large-Scale Study of Relevance Assessments with Large Language Models: An Initial Lookcited by 18
  34. 2024Can We Use Large Language Models to Fill Relevance Judgment Holes?cited by 17
  35. 2025LLMs can be Fooled into Labelling a Document as Relevant (best caf'e near me; this paper is perfectly relevant)cited by 10
  36. 2024Reliable Confidence Intervals for Information Retrieval Evaluation Using Generative A.Icited by 9
  37. 2024SynDL: A Large-Scale Synthetic Test Collection for Passage Retrievalcited by 9
  38. 2024Large Language Models for Relevance Judgment in Product Searchcited by 8

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 9.

15 papers.

Sort:

By area:

  1. 2025When Large Language Models are Reliable for Judging Empathic Communicationcited by 21
  2. 2025Beyond Contrastive Learning: Synthetic Data Enables List-wise Training with Multiple Levels of Relevancecited by 4
  3. 2025Beyond GeneGPT: A Multi-Agent Architecture with Open-Source LLMs for Enhanced Genomic Question Answeringcited by 3
  4. 2025Uncovering the Limitations of Query Performance Prediction: Failures, Insights, and Implications for Selective Query Processingcited by 2
  5. 2025Mitigating the Threshold Priming Effect in Large Language Model-Based Relevance Judgments via Personality Infusingcited by 2
  6. 2026Query-Document Dense Vectors for LLM Relevance Judgment Bias Analysiscited by 1
  7. 2026Judging with Personality and Confidence: A Study on Personality-Conditioned LLM Relevance Assessmentcited by 1
  8. 2026Insider Knowledge: How Much Can RAG Systems Gain from Evaluation Secrets?cited by 1
  9. 2026Understanding LLM Evaluator Behavior: A Structured Multi-Evaluator Framework for Merchant Risk Assessmentcited by 1
  10. 2026Persona Conditioning as an Assessor-Sensitivity Probe for LLM-Based IR Evaluationcited by 1
  11. 2025The Effect of Document Summarization on LLM-Based Relevance Judgmentscited by 0
  12. 2026Pro-AI Bias in Large Language Modelscited by 0
3 more
  1. 2026RMIT-ADM+S at the MMU-RAG NeurIPS 2025 Competitioncited by 0
  2. 2026Localization Boosting for Growth Markets: Mitigating Cross-Locale Behavioral Bias in Learning-to-Rankcited by 0
  3. 2026Free-Text Evaluation of LLMs for 5G Domain Knowledge and Fault Analysis using LLM-as-Judgecited by 0