mlsanityData to 14 Oct 2026

Contrastive Decoding Mitigates Score Range Bias in LLM-as-a-Judge

arXiv 2510.18196 ↗First posted 21 Oct 2025cs.CL cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) are commonly used as evaluators in various applications, but the reliability of the outcomes remains a challenge. One such challenge is using LLMs-as-judges for direct assessment, i.e., assigning scores from a specified range without any references. Focusing on summarization, we first show that this challenge stems from LLM judge outputs being associated with score range bias, i.e., LLM judge outputs are highly sensitive to pre-defined score ranges. We also show that similar biases exist among models from the same family. We then mitigate this bias through contrastive decoding, achieving up to 11.7% relative improvement on average in Spearman correlation with human judgments across different score ranges.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 19 of its 26 references that are in the map. First posted on arXiv 21 Oct 2025.

Builds on

19 of 26 references.

References · 19

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2024Qwen2.5 Technical Reportcited by 4,942
  3. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  4. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  5. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  6. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  7. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  8. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
  9. 2020SummEval: Re-evaluating Summarization Evaluationcited by 674
  10. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 609
  11. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 446
  12. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 368
7 more
  1. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 367
  2. 2024Self-Preference Bias in LLM-as-a-Judgecited by 258
  3. 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluatorscited by 149
  4. 2021Investigating the Limitations of Transformers with Simple Arithmetic Taskscited by 139
  5. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 93
  6. 2025Great Models Think Alike and this Undermines AI Oversightcited by 58
  7. 2024Language Models Do Hard Arithmetic Tasks Easily and Hardly Do Easy Arithmetic Taskscited by 24

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026GeoRC: A Benchmark for Geolocation Reasoning Chainscited by 4
  2. 2026PaperJury: Due-Process Review for Bounded LaTeX Revisioncited by 1
  3. 2026Mitigating Scoring Bias in LLM-as-a-Judge via Random Number Generationcited by 0