# Evaluating and Mitigating LLM-as-a-judge Bias in Communication Systems

arXiv: https://arxiv.org/abs/2510.12462 · HTML: https://mlsanity.com/paper/2510.12462/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 28 references that are in the map.
- First posted on arXiv 14 Oct 2025.

## Builds on (most cited first, 14 of 14)

- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods](https://mlsanity.com/paper/2412.05579/) (2024)
- [Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge](https://mlsanity.com/paper/2410.02736/) (2024)
- [JudgeLM: Fine-tuned Large Language Models are Scalable Judges](https://mlsanity.com/paper/2310.17631/) (2023)
- [Humans or LLMs as the Judge? A Study on Judgement Biases](https://mlsanity.com/paper/2402.10669/) (2024)
- [JudgeBench: A Benchmark for Evaluating LLM-based Judges](https://mlsanity.com/paper/2410.12784/) (2024)
- [Transformers for Modeling Physical Systems](https://mlsanity.com/paper/2010.03957/) (2020)
- [Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templates](https://mlsanity.com/paper/2408.13006/) (2024)
- [Mitigating the Bias of Large Language Model Evaluation](https://mlsanity.com/paper/2409.16788/) (2024)
- [LLM-Evaluation Tropes: Perspectives on the Validity of LLM-Evaluations](https://mlsanity.com/paper/2504.19076/) (2025)
- [Intent Assurance using LLMs guided by Intent Drift](https://mlsanity.com/paper/2402.00715/) (2024)
- [CompassJudger-2: Towards Generalist Judge Model via Verifiable Rewards](https://mlsanity.com/paper/2507.09104/) (2025)
- [Towards End-to-End Network Intent Management with Large Language Models](https://mlsanity.com/paper/2504.13589/) (2025)

## Built on by (most cited first, 6 of 6)

- [MM-JudgeBias: A Benchmark for Evaluating Compositional Biases in MLLM-as-a-Judge](https://mlsanity.com/paper/2604.18164/) (2026)
- [An LLM-Native Psychometric Instrument Reveals a Self-Report--Behavior Gap Across 25 Models](https://mlsanity.com/paper/2606.09843/) (2026)
- [Scalable and Reliable Evaluation of AI Knowledge Retrieval Systems: RIKER and the Coherent Simulated Universe](https://mlsanity.com/paper/2601.08847/) (2025)
- [Mitigating Translationese Bias in Multilingual LLM-as-a-Judge via Disentangled Information Bottleneck](https://mlsanity.com/paper/2603.10351/) (2026)
- [Stability vs. Manipulability: Evaluating Robustness Under Post-Decision Interaction in LLM Judges](https://mlsanity.com/paper/2606.05384/) (2026)
- [Enjoy Your Talk: A Human-Centered Benchmark for Multi-Turn Dialogue with Decoupled User Simulation, Target Modeling, and Judging](https://mlsanity.com/paper/2607.10428/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are increasingly being used to autonomously evaluate the quality of content in communication systems, e.g., to assess responses in telecom customer support chatbots. However, the impartiality of these AI "judges" is not guaranteed, and any biases in their evaluation criteria could skew outcomes and undermine user trust. In this paper, we systematically investigate judgment biases across 6 LLM-as-a-judge models spanning both prompt-based and fine-tuned judges under the pointwise scoring setting, encompassing 11 types of biases that cover both implicit and explicit forms. We observed that state-of-the-art LLM judges demonstrate robustness to biased inputs, generally assigning them lower scores than the corresponding clean samples. We further found that fine-tuning an LLM on high-scoring yet biased responses can significantly degrade its performance, highlighting the risk of training on biased data. We also discovered that the judged scores correlate with task difficulty: a challenging dataset like GPQA yields lower average scores, whereas an open-ended reasoning dataset (e.g., JudgeLM-val) sees higher average scores. Finally, we proposed four potential mitigation strategies to ensure fair and reliable AI judging in practical communication scenarios.