mlsanityData to 7 Oct 2026

When Languages Disagree: Self-Evolving Multilingual LLM Judges

arXiv 2606.08092 ↗First posted 6 Jun 2026cs.CLcited by 2

On the map: Synthetic data and self-improvement for LLMs → Self-rewarding alignment and self-training

Abstract (arXiv)

Multilingual LLM-as-a-judge is widely used to evaluate model outputs across languages, but suffers from cross-lingual inconsistency (Fu and Liu, 2025). Existing methods typically treat this inconsistency as noise and mitigate it through voting or aggregation. In this work, we instead show that multilingual inconsistency can provide complementary evaluation signals. Our oracle analysis finds that sampling judgments across languages yields a higher performance upper bound than single-language judging, indicating that different languages potentially include complementary judgments. Motivated by this finding, we propose SEMJ, a self-evolving multilingual judge that leverages cross-lingual inconsistency for iterative refinement. SEMJ constructs multilingual variants of each input, collects independent judgments and rationales, and feeds inconsistent outputs back for self-reflection and re-evaluation. Experiments on multiple benchmarks show that SEMJ consistently outperforms voting and reflection baselines in both accuracy and cross-lingual consistency. Further analysis shows that inconsistency triggers useful re-evaluation, which improves judgment quality.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Self-rewarding alignment and self-training, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 30 of its 45 references that are in the map. First posted on arXiv 6 Jun 2026.

Builds on

30 of 45 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024A Survey on LLM Inference-Time Self-Improvement18 Dec
  2. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge28 Jul

Other references · 28

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  5. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  6. 2024A Survey on LLM-as-a-Judgecited by 1,384
  7. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  8. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
  9. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  10. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  11. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  12. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
16 more
  1. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 351
  2. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 330
  3. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 311
  4. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 301
  5. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 300
  6. 2025A Comprehensive Survey of Self-Evolving AI Agents: A New Paradigm Bridging Foundation Models and Lifelong Agentic Systemscited by 177
  7. 2023Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Modelscited by 117
  8. 2023Cross-lingual Prompting: Improving Zero-shot Chain-of-Thought Reasoning across Languagescited by 96
  9. 2025Lost in Multilinguality: Dissecting Cross-lingual Factual Inconsistency in Transformer Language Modelscited by 43
  10. 2024Auto-Arena: Automating LLM Evaluations with Agent Peer Battles and Committee Discussionscited by 40
  11. 2025Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworkscited by 37
  12. 2025Improving LLM-as-a-Judge Inference with the Judgment Distributioncited by 36
  13. 2025How Reliable is Multilingual LLM-as-a-Judge?cited by 29
  14. 2024Self-rationalization improves LLM as a fine-grained judgecited by 25
  15. 2025CALM: Unleashing the Cross-Lingual Self-Aligning Ability of Language Model Question Answeringcited by 10
  16. 2025Do Before You Judge: Self-Reference as a Pathway to Better LLM Evaluationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loopscited by 25
  2. 2026Where's Waldo? Query-language Preference under Cross-lingual Knowledge Disparitiescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026JudgeArena: A Unified Framework for Reproducible LLM-Judge Evaluation
  2. 2025Leveraging LLMs as Meta-Judges: A Multi-Agent Framework for Evaluating LLM Judgments
  3. 2024MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Models
  4. 2026When Consistency Does Not Mean Reliability: Evaluating Local LLM Judges Against Human Ratings
  5. 2025Judge as A Judge: Improving the Evaluation of Retrieval-Augmented Generation through the Judge-Consistency of Large Language Models
  6. 2026Lower-Resource, Higher Scores: Language Bias in LLM Evaluators
  7. 2026BabelJudge: Measuring LLM-as-a-Judge Reliability Across Languages and Agent Trajectories
  8. 2025Cross-Lingual Consistency: A Novel Inference Framework for Advancing Reasoning in Large Language Models