mlsanityData to 14 Oct 2026

Rank Reversal in Multilingual LLM Judges: A Label-Free Double-Centering Calibrator

arXiv 2608.22432 ↗First posted 23 Aug 2026cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Multilingual LLM judges produce different evaluator-backbone rankings depending on the prompt language: on an eight-language Agent-as-a-Judge benchmark, the top-ranked backbone alternates across English, Arabic, Chinese, Hindi, Japanese, Spanish, Turkish, and Swahili, and 7 of 15 backbone pairs show statistically significant pairwise rank reversal. We treat this as a measurement problem. The multilingual judge score decomposes additively into task difficulty, backbone skill, and a language-backbone interaction term, the last of which is recoverable without human labels by double-centering the cell-mean score matrix. We make this estimator (Consensus-Based Calibration, CBC) explicit, give an O(1/√n) finite-sample concentration bound with variance constant (1-tfrac1m)(1-tfrac1k), and show that it is unbiased even when task-language interactions are present. Across 7,920 judge runs (6 backbones, 8 languages, 55 tasks, 3 frameworks), CBC raises held-out cross-task rank consistency τ from 0.650 to 0.902 and agrees with the held-out additive-model oracle in 100% of per-language decisions versus 68.5% raw; these are consistency diagnostics, not human-grounded correctness measures. On a separately collected M-RewardBench panel (7 languages, 1,500 items per language, 10,500 language-item instances, 5 evaluators), panel agreement with the public human gold preferences rises from 68.7% to 76.6% (gain 7.9 percentage points, 95% CI [6.0, 9.9]), our strongest external evidence of downstream usefulness. The estimator is the standard two-way ANOVA interaction-recovery operation under sum-to-zero contrasts; our contribution is its application as a label-free post-hoc calibrator for multilingual LLM judges, an explicit finite-sample concentration bound, and an unbiasedness result that holds even under task-language misspecification.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 17 of its 29 references that are in the map. First posted on arXiv 23 Aug 2026.

Builds on

17 of 29 references.

References · 17

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  3. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 318
  4. 2024tinyBenchmarks: evaluating LLMs with fewer examplescited by 310
  5. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 243
  6. 2024Agent-as-a-Judge: Evaluate Agents with Agentscited by 238
  7. 2024Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluationcited by 191
  8. 2023Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?cited by 88
  9. 2023SemEval-2023 Task 11: Learning With Disagreements (LeWiDi)cited by 52
  10. 2024MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Modelscited by 35
  11. 2024CalibraEval: Calibrating Prediction Distribution to Mitigate Selection Bias in LLMs-as-Judgescited by 34
  12. 2025How Reliable is Multilingual LLM-as-a-Judge?cited by 30
5 more
  1. 2024Cross-Lingual Auto Evaluation for Assessing Multilingual LLMscited by 18
  2. 2019Truth Inference at Scale: A Bayesian Model for Adjudicating Highly Redundant Crowd Annotationscited by 11
  3. 2026A Judge-Aware Ranking Framework for Evaluating Large Language Models without Ground Truthcited by 7
  4. 2026Cross-Lingual LLM-Judge Transfer via Evaluation Decompositioncited by 3
  5. 2026Multilingual Prompt Localization for Agent-as-a-Judge: Language and Backbone Sensitivity in Requirement-Level Evaluationcited by 2

Built on by

Not cited yet.