mlsanityData to 7 Oct 2026

BabelJudge: Measuring LLM-as-a-Judge Reliability Across Languages and Agent Trajectories

arXiv 2606.22329 ↗First posted 21 Jun 2026cs.CL cs.AIcited by 2

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

LLM-as-a-judge has become the dominant approach to scalable evaluation in NLP pipelines, yet judges themselves carry systematic biases that raw accuracy hides: they favor responses placed in slot A (position bias), they prefer longer responses regardless of quality (verbosity bias), and their reliability degrades sharply in lower-resource languages. We introduce BabelJudge, an open-source benchmark and reliability audit framework that measures all four failure modes -- position bias, verbosity bias, order inconsistency, and cross-lingual degradation -- on any judge model, without requiring human preference labels. The key insight is gold-labelling by degradation: starting from a high-quality reference response and applying a controlled perturbation yields a pairwise item whose gold label is known by construction, eliminating annotation cost. We evaluate Qwen2.5-7B-Instruct-4bit across English, Hindi, Arabic, and Swahili and find that our composite bias-penalised reliability score drops from 0.714 in Hindi to 0.550 in Swahili, a gap that raw accuracy (0.835 vs. 0.660) understates. Swahili order consistency collapses to 0.480, meaning judge verdicts are near-random under slot-order swaps -- a failure mode invisible to accuracy alone. We further extend the framework to agentic evaluation via nine trajectory-level perturbations (argument corruption, tool swaps, hallucinated calls, missing steps) and three new metrics: tool accuracy, hallucination detection rate, and trajectory-length bias. BabelJudge is released as a Python package supporting 11 judge backends. Code: https://github.com/Shreyaskc/BabelJudge

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 16 of its 21 references that are in the map. First posted on arXiv 21 Jun 2026.

Builds on

16 of 21 references.

References · 16

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2022Red Teaming Language Models with Language Modelscited by 983
  4. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  5. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  6. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
  7. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 454
  8. 2023GPTScore: Evaluate as You Desirecited by 357
  9. 2023MEGA: Multilingual Evaluation of Generative AIcited by 291
  10. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 285
  11. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 200
  12. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 174
4 more
  1. 2023Verbosity Bias in Preference Labeling by Large Language Modelscited by 89
  2. 2020Look at the First Sentence: Position Bias in Question Answeringcited by 83
  3. 2023Calibrating LLM-Based Evaluatorcited by 53
  4. 2020Evaluating Models' Local Decision Boundaries via Contrast Setscited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026OmnilingualGAIA2: Evaluating the Multilingual Gap in Frontier AI Agentscited by 2
  2. 2026Uncertainty Quantification for LLM Agents: A Taxonomy, an Evaluation Protocol, and an Empirical Studycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026JudgeArena: A Unified Framework for Reproducible LLM-Judge Evaluation
  2. 2024CompassJudger-1: All-in-one Judge Model Helps Model Evaluation and Evolution
  3. 2026A Judge-Aware Ranking Framework for Evaluating Large Language Models without Ground Truth
  4. 2026LLM as a Meta-Judge: Synthetic Data for NLP Evaluation Metric Validation
  5. 2025Evaluating Scoring Bias in LLM-as-a-Judge
  6. 2026When Consistency Does Not Mean Reliability: Evaluating Local LLM Judges Against Human Ratings
  7. 2025Neither Valid nor Reliable? Investigating the Use of LLMs as Judges
  8. 2024Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges