mlsanityData to 7 Oct 2026

BiasScope: Towards Automated Detection of Bias in LLM-as-a-Judge Evaluation

arXiv 2602.09383 ↗First posted 10 Feb 2026cs.CL cs.AI cs.SEcited by 12

Not on the map. Why?

Abstract (arXiv)

LLM-as-a-Judge has been widely adopted across various research and practical applications, yet the robustness and reliability of its evaluation remain a critical issue. A core challenge it faces is bias, which has primarily been studied in terms of known biases and their impact on evaluation outcomes, while automated and systematic exploration of potential unknown biases is still lacking. Nevertheless, such exploration is crucial for enhancing the robustness and reliability of evaluations. To bridge this gap, we propose BiasScope, a LLM-driven framework for automatically and at scale discovering potential biases that may arise during model evaluation. BiasScope can uncover potential biases across different model families and scales, with its generality and effectiveness validated on the JudgeBench dataset. It overcomes the limitations of existing approaches, transforming bias discovery from a passive process relying on manual effort and predefined bias lists into an active and comprehensive automated exploration. Moreover, based on BiasScope, we propose JudgeBench-Pro, an extended version of JudgeBench and a more challenging benchmark for evaluating the robustness of LLM-as-a-judge. Strikingly, even powerful LLMs as evaluators show error rates above 50% on JudgeBench-Pro, underscoring the urgent need to strengthen evaluation robustness and to mitigate potential biases further.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 24 of its 35 references that are in the map. First posted on arXiv 10 Feb 2026.

Builds on

24 of 35 references.

References · 24

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  4. 2024A Survey on LLM-as-a-Judgecited by 1,384
  5. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  6. 2024Self-Rewarding Language Modelscited by 654
  7. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  8. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
  9. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 314
  10. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 294
  11. 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 249
  12. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 236
12 more
  1. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 168
  2. 2023Benchmarking Cognitive Biases in Large Language Models as Evaluatorscited by 147
  3. 2024Optimization-based Prompt Injection Attack to LLM-as-a-Judgecited by 123
  4. 2025Preference Leakage: A Contamination Problem in LLM-as-a-judgecited by 119
  5. 2024Pride and Prejudice: LLM Amplifies Self-Bias in Self-Refinementcited by 111
  6. 2023Split and Merge: Aligning Position Biases in LLM-based Evaluatorscited by 71
  7. 2024Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templatescited by 71
  8. 2023ICE-Score: Instructing Large Language Models to Evaluate Codecited by 50
  9. 2024Thinking LLMs: General Instruction Following with Thought Generationcited by 45
  10. 2025Solver-Informed RL: Grounding Large Language Models for Authentic Optimization Modelingcited by 44
  11. 2025Beyond the Surface: Measuring Self-Preference in LLM Judgmentscited by 30
  12. 2025Do LLMs have a Gender (Entropy) Bias?cited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 12.

12 papers.

  1. 2026JudgeArena: A Unified Framework for Reproducible LLM-Judge Evaluationcited by 2
  2. 2026PaperJury: Due-Process Review for Bounded LaTeX Revisioncited by 1
  3. 2026Coordinates of Capability: A Unified MTMM-Geometric Framework for LLM Evaluationcited by 0
  4. 2026Are LLMs Bad at Moral Reasoning?cited by 0
  5. 2026Data-Centric Benchmarking of Exploit Generation in LLMs: Understanding the Impact of Fine-Tuningcited by 0
  6. 2026Beyond Static Rules: Automated Discovery of Latent Vulnerabilities in Text-to-SQLcited by 0
  7. 2026Automatic Hard Example Synthesis with Multi-Level Agentic Data Curationcited by 0
  8. 2026Who Verifies the Benchmark? Decentralizing Trust in Large Language Model Evaluationcited by 0
  9. 2026AlignDiff: Exploiting Model-Intrinsic Information for Better Preference Data Selectioncited by 0
  10. 2026UniRRM: Unified Reasoning Reward Models Across Languages and Evaluation Paradigmscited by 0
  11. 2026Debiasing as a Measurement Intervention: Calibrated Ties and Resolution Loss in LLM-as-a-Judge Evaluationcited by 0
  12. 2026Bad Genius: Counterfactual-Guided Harness Evolution Beyond Task-Specific Shortcutscited by 0