mlsanityData to 14 Oct 2026

Preference Leakage: A Contamination Problem in LLM-as-a-judge

arXiv 2502.01534 ↗First posted 3 Feb 2025cs.LG cs.AI cs.CLcited by 125

Not on the map. Why?

Cited by 22 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large Language Models (LLMs) as judges and LLM-based data synthesis have emerged as two fundamental LLM-driven data annotation methods in model development. While their combination significantly enhances the efficiency of model training and evaluation, little attention has been given to the potential contamination brought by this new model development paradigm. In this work, we expose preference leakage, a contamination problem in LLM-as-a-judge caused by the relatedness between the synthetic data generators and LLM-based evaluators. To study this issue, we first define three common relatednesses between the data generator LLM and the judge LLM: being the same model, having an inheritance relationship, and belonging to the same model family. Through extensive experiments, we empirically confirm the bias of judges towards their related student models caused by preference leakage across multiple LLM baselines and benchmarks. Further analysis suggests that preference leakage is a pervasive and real-world problem that is harder to detect compared to previously identified biases in LLM-as-a-judge scenarios. All of these findings imply that preference leakage is a widespread and challenging problem in the area of LLM-as-a-judge. We release all codes and data at: https://github.com/David-Li0406/Preference-Leakage.

Key facts. Cited by 125 papers in the map (data through 14 Oct 2026). Builds on 68 of its 98 references that are in the map. First posted on arXiv 3 Feb 2025.

Builds on

68 of 98 references.

References · 68

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  6. 2024Qwen2.5 Technical Reportcited by 4,942
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  8. 2023Mistral 7Bcited by 3,164
  9. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  10. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  11. 2023AgentBench: Evaluating LLMs as Agentscited by 1,267
  12. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
38 more
  1. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  2. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
  3. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
  4. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
  5. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  6. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
  7. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 492
  8. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 446
  9. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  10. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 368
  11. 2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuningcited by 364
  12. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
  13. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 323
  14. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
  15. 2024Self-Preference Bias in LLM-as-a-Judgecited by 258
  16. 2024Agent-as-a-Judge: Evaluate Agents with Agentscited by 238
  17. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 229
  18. 2023MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Usecited by 217
  19. 2024Large Language Models for Data Annotation and Synthesis: A Surveycited by 205
  20. 2024Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judgescited by 205
  21. 2023Time Travel in LLMs: Tracing Data Contamination in Large Language Modelscited by 177
  22. 2023Benchmarking Cognitive Biases in Large Language Models as Evaluatorscited by 150
  23. 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 147
  24. 2024Optimization-based Prompt Injection Attack to LLM-as-a-Judgecited by 143
  25. 2023Investigating Data Contamination in Modern Benchmarks for Large Language Modelscited by 138
  26. 2023Human-like Summarization Evaluation with ChatGPTcited by 126
  27. 2024Generalization or Memorization: Data Contamination and Trustworthy Evaluation for Large Language Modelscited by 122
  28. 2024Pride and Prejudice: LLM Amplifies Self-Bias in Self-Refinementcited by 114
  29. 2023Task Contamination: Language Models May Not Be Few-Shot Anymorecited by 99
  30. 2024Bias and Unfairness in Information Retrieval Systems: New Challenges in the LLM Eracited by 98
  31. 2024Investigating Data Contamination for Pre-training Language Modelscited by 91
  32. 2023LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scorescited by 90
  33. 2024Self-Alignment for Factuality: Mitigating Hallucinations in LLMs via Self-Evaluationcited by 80
  34. 2024MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixturescited by 74
  35. 2021Memorization vs. Generalization: Quantifying Data Leakage in NLP Performance Evaluationcited by 70
  36. 2024Can LLMs Learn from Previous Mistakes? Investigating LLMs' Errors to Boost for Reasoningcited by 69
  37. 2023AlignBench: Benchmarking Chinese Alignment of Large Language Modelscited by 68
  38. 2024Selective Reflection-Tuning: Student-Selected Data Recycling for LLM Instruction-Tuningcited by 57

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 60; 2026: 65.

125 papers.

Sort:

By area:

  1. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 320
  2. 2025MCP-Universe: Benchmarking Large Language Models with Real-World Model Context Protocol Serverscited by 96
  3. 2025One Token to Fool LLM-as-a-Judgecited by 60
  4. 2025xVerify: Efficient Answer Verifier for Reasoning Model Evaluationscited by 55
  5. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  6. 2025SophiaVL-R1: Reinforcing MLLMs Reasoning with Thinking Rewardcited by 45
  7. 2025The relationship between reasoning and performance in large language models--o3 (mini) thinks harder, not longercited by 41
  8. 2025Neither Valid nor Reliable? Investigating the Use of LLMs as Judgescited by 39
  9. 2025Do LLM Evaluators Prefer Themselves for a Reason?cited by 38
  10. 2025ResearchCodeBench: Benchmarking LLMs on Implementing Novel Machine Learning Research Codecited by 36
  11. 2025A Survey on Data Contamination for Large Language Modelscited by 35
  12. 2025Beyond the Surface: Measuring Self-Preference in LLM Judgmentscited by 31
38 more
  1. 2025DRP: Distilled Reasoning Pruning with Skill-aware Step Decomposition for Efficient Large Reasoning Modelscited by 30
  2. 2025A Survey on Privacy Risks and Protection in Large Language Modelscited by 21
  3. 2025TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Themcited by 20
  4. 2025Understanding and Mitigating Overrefusal in LLMs from an Unveiling Perspective of Safety Decision Boundarycited by 18
  5. 2026WebCoderBench: Benchmarking Web Application Generation with Comprehensive and Interpretable Evaluation Metricscited by 18
  6. 2026Efficient Evaluation of LLM Performance with Statistical Guaranteescited by 18
  7. 2025YESciEval: Robust LLM-as-a-Judge for Scientific Question Answeringcited by 16
  8. 2025CodeIF-Bench: Evaluating Instruction-Following Capabilities of Large Language Models in Interactive Code Generationcited by 15
  9. 2025Beyond the Surface: Enhancing LLM-as-a-Judge Alignment with Human via Internal Representationscited by 15
  10. 2025Krikri: Advancing Open Large Language Models for Greekcited by 14
  11. 2025DOGe: Defensive Output Generation for LLM Protection Against Knowledge Distillationcited by 14
  12. 2026BiasScope: Towards Automated Detection of Bias in LLM-as-a-Judge Evaluationcited by 14
  13. 2025LongWriter-Zero: Mastering Ultra-Long Text Generation via Reinforcement Learningcited by 13
  14. 2026Generative Visual Code Mobile World Modelscited by 13
  15. 2026CARE: Confounder-Aware Aggregation for Reliable LLM Evaluationcited by 13
  16. 2026Bridging Reasoning Trajectories in On-Policy Distillation via Near-Future Guidancecited by 13
  17. 2025Consensus Entropy: Harnessing Multi-VLM Agreement for Self-Verifying and Self-Improving OCRcited by 12
  18. 2025What Matters in Data for DPO?cited by 12
  19. 2025Adaptive Prediction-Powered AutoEval with Reliability and Efficiency Guaranteescited by 11
  20. 2025Judging with Many Minds: Do More Perspectives Mean Less Prejudice? On Bias Amplifications and Resistance in Multi-Agent Based LLM-as-Judgecited by 11
  21. 2026Self-Preference Bias in Rubric-Based Evaluation of Large Language Modelscited by 11
  22. 2026Cornerstones or Stumbling Blocks? Deciphering the Rock Tokens in On-Policy Distillationcited by 11
  23. 2025CLIPPER: Compression enables long-context synthetic data generationcited by 10
  24. 2025DCR: Quantifying Data Contamination in LLMs Evaluationcited by 10
  25. 2026DeepResearch-9K: A Challenging Benchmark Dataset of Deep-Research Agentcited by 10
  26. 2026Quantifying and Mitigating Self-Preference Bias of LLM Judgescited by 10
  27. 2025What do Large Language Models Say About Animals? Investigating Risks of Animal Harm in Generated Textcited by 9
  28. 2025The Quest for Efficient Reasoning: A Data-Centric Benchmark to CoT Distillationcited by 9
  29. 2025Lexical Hints of Accuracy in LLM Reasoning Chainscited by 9
  30. 2025Who's Your Judge? On the Detectability of LLM-Generated Judgmentscited by 8
  31. 2025Beyond Accuracy: The Role of Calibration in Self-Improving Large Language Modelscited by 7
  32. 2025CHARM: Calibrating Reward Models With Chatbot Arena Scorescited by 7
  33. 2025Self-Improvement of Language Models by Post-Training on Multi-Agent Debatecited by 7
  34. 2025Self-Improving VLM Judges Without Human Annotationscited by 7
  35. 2026Remembering More, Risking More: Longitudinal Safety Risks in Memory-Equipped LLM Agentscited by 7
  36. 2025Quantitative LLM Judgescited by 6
  37. 2026Seven simple steps for log analysis in AI systemscited by 6
  38. 2025Retrieval is Not Enough: Enhancing RAG Reasoning through Test-Time Critique and Optimizationcited by 5

50 most cited of 125 All 125 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026JudgeArena: A Unified Framework for Reproducible LLM-Judge Evaluation
  2. 2024Dissecting Human and LLM Preferences
  3. 2026Automated Concept Discovery for LLM-as-a-Judge Preference Analysis
  4. 2025LLMs Cannot Reliably Judge (Yet?): A Comprehensive Assessment on the Robustness of LLM-as-a-Judge
  5. 2025How Humans Help LLMs: Assessing and Incentivizing Human Preference Annotators
  6. 2025Bias Testing and Mitigation in Black Box LLMs using Metamorphic Relations
  7. 2025CLEAR: Error Analysis via LLM-as-a-Judge Made Easy
  8. 2024Direct Judgement Preference Optimization