mlsanityData to 14 Oct 2026

Chain-of-Models: Cross-Model Auditing for Bias-Robust LLM Judges

arXiv 2607.28636 ↗First posted 19 May 2026cs.CL cs.CYcited by 0

Not on the map. Why?

Abstract (arXiv)

LLMs increasingly serve as automated judges, but their judgments remain vulnerable to cognitive biases. Existing mitigations mostly rely on prompt-driven debiasing, which is brittle across bias types, or human evaluation, which does not scale. We study Chain-of-Models (CoM), an automated audit pipeline in which a second model inspects the first model's reasoning trace before producing the final judgment. The key design question is whether the auditor should be the same model, a same-family model, or a different-family model. Across 9 models from 6 families, 4 cognitive biases, and 4 factual datasets, we find that auditor identity matters in two ways. First, standalone bias resistance does not predict audit effectiveness: Kimi-K2.5 is the strongest standalone model on several biases, yet is a weak auditor for Qwen2.5-72B's biased traces. Second, the best auditor is bias-specific: GPT-4o is strongest on bandwagon, authority, and distraction, while GLM-5 is strongest on sycophancy. We operationalize these findings with a per-bias auditor selection rule that, given the bias type, scores candidates along functional diversity, per-bias standalone resistance, and calibrated audit effectiveness. Under a calibration/test split, the selector reaches the highest accuracy across the four biased slices (0.884 vs. 0.824 for the strongest single fixed auditor and 0.805 for the no-audit baseline). We release data, configurations, and an LLM-agent skill at https://anonymous.4open.science/r/chain-of-models-B585 .

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 25 of its 37 references that are in the map. First posted on arXiv 19 May 2026.

Builds on

25 of 37 references.

References · 25

  1. 2023Let's Verify Step by Stepcited by 3,789
  2. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  3. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  4. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  5. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  6. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  7. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 1,011
  8. 2023Teaching Large Language Models to Self-Debugcited by 988
  9. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
  10. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
  11. 2024Mixture-of-Agents Enhances Large Language Model Capabilitiescited by 438
  12. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 368
13 more
  1. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 326
  2. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 318
  3. 2024Self-Preference Bias in LLM-as-a-Judgecited by 258
  4. 2024Large Language Models are Inconsistent and Biased Evaluatorscited by 153
  5. 2024Optimization-based Prompt Injection Attack to LLM-as-a-Judgecited by 143
  6. 2024Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessmentcited by 126
  7. 2025Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judgecited by 78
  8. 2025One Token to Fool LLM-as-a-Judgecited by 60
  9. 2024LLM-TOPLA: Efficient LLM Ensemble by Maximising Diversitycited by 42
  10. 2025Assessing Judging Bias in Large Reasoning Models: An Empirical Studycited by 30
  11. 2025LLM DNA: Tracing Model Evolution via Functional Representationscited by 15
  12. 2025Any Large Language Model Can Be a Reliable Judge: Debiasing with a Reasoning-based Bias Detectorcited by 14
  13. 2026PeerRank: Autonomous LLM Evaluation Through Web-Grounded, Bias-Controlled Peer Reviewcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Quantifying and Mitigating Self-Preference Bias of LLM Judges
  2. 2025Safer or Luckier? LLMs as Safety Evaluators Are Not Robust to Artifacts
  3. 2026Autorubric: A Unifying Framework for Rubric-Based LLM Evaluation on Non-Verifiable Tasks
  4. 2026White-Box Sensitivity Auditing with Steering Vectors
  5. 2026Biases in the Blind Spot: Detecting What LLMs Fail to Mention
  6. 2026Toward Robust LLM-Based Judges: Taxonomic Bias Evaluation and Debiasing Optimization
  7. 2026LLMs as a Jury: Cross-Model Consensus Can Outperform Process Reward Models for LLM Reasoning
  8. 2026Audit Without Verification: When LLM Accountability Layers Relay Rather Than Check