mlsanityData to 7 Oct 2026

CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation

arXiv 2603.00039 ↗First posted 9 Feb 2026cs.LG cs.AI stat.MLcited by 10

Not on the map. Why?

Abstract (arXiv)

LLM-as-a-judge ensembles are the standard paradigm for scalable evaluation, but their aggregation mechanisms suffer from a fundamental flaw: they implicitly assume that judges provide independent estimates of true quality. However, in practice, LLM judges exhibit correlated errors caused by shared latent confounders -- such as verbosity, stylistic preferences, or training artifacts -- causing standard aggregation rules like majority vote or averaging to provide little gain or even amplify systematic mistakes. To address this, we introduce CARE, a confounder-aware aggregation framework that explicitly models LLM judge scores as arising from both a latent true-quality signal and shared confounding factors. Rather than heuristically re-weighting judges, CARE separates quality from confounders without access to ground-truth labels. We provide theoretical guarantees for identifiability and finite-sample recovery under shared confounders, and we quantify the systematic bias incurred when aggregation models omit confounding latent factors. Across 12 public benchmarks spanning continuous scoring, binary classification, and pairwise preference settings, CARE improves aggregation accuracy, reducing error by up to 26.8%. Code is released in hrefhttps://github.com/SprocketLab/CAREhttps://github.com/SprocketLab/CARE.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 49 of its 72 references that are in the map. First posted on arXiv 9 Feb 2026.

Builds on

49 of 72 references.

References · 49

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2024GPT-4o System Cardcited by 4,467
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2020Learning to summarize from human feedbackcited by 2,562
  7. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  8. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  9. 2025Gemma 3 Technical Reportcited by 1,686
  10. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  11. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  12. 2024Yi: Open Foundation Models by 01.AIcited by 706
37 more
  1. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 491
  2. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  3. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  4. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 416
  5. 2019Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classificationcited by 396
  6. 2017Snorkel: Rapid Training Data Creation with Weak Supervisioncited by 370
  7. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
  8. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 351
  9. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 314
  10. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 311
  11. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 301
  12. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  13. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  14. 2024Debating with More Persuasive LLMs Leads to More Truthful Answerscited by 255
  15. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 252
  16. 2025Crossing the Reward Bridge: Expanding RL with Verifiable Rewards Across Diverse Domainscited by 150
  17. 2024CycleResearcher: Improving Automated Research via Automated Reviewcited by 142
  18. 2025Preference Leakage: A Contamination Problem in LLM-as-a-judgecited by 119
  19. 2023PRD: Peer Rank and Discussion Improve Large Language Model based Evaluationscited by 114
  20. 2020ASSET: A Dataset for Tuning and Evaluation of Sentence Simplification Models with Multiple Rewriting Transformationscited by 101
  21. 2020Fast and Three-rious: Speeding Up Weak Supervision with Triplet Methodscited by 98
  22. 2024OffsetBias: Leveraging Debiased Data for Tuning Evaluatorscited by 94
  23. 2023Split and Merge: Aligning Position Biases in LLM-based Evaluatorscited by 71
  24. 2023Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarizationcited by 70
  25. 2025One Token to Fool LLM-as-a-Judgecited by 56
  26. 2018Snorkel DryBell: A Case Study in Deploying Weak Supervision at Industrial Scalecited by 52
  27. 2022On the Limitations of Reference-Free Evaluations of Generated Textcited by 52
  28. 2025Improving LLM-as-a-Judge Inference with the Judgment Distributioncited by 36
  29. 2021End-to-End Weak Supervisioncited by 33
  30. 2024CalibraEval: Calibrating Prediction Distribution to Mitigate Selection Bias in LLMs-as-Judgescited by 32
  31. 2021Universalizing Weak Supervisioncited by 27
  32. 2022Using Interactive Feedback to Improve the Accuracy and Explainability of Question Answering Systems Post-Deploymentcited by 25
  33. 2019Multi-Resolution Weak Supervision for Sequential Datacited by 17
  34. 2024The ALCHEmist: Automated Labeling 500x CHEaper Than LLM Data Annotatorscited by 10
  35. 2024JudgeBlender: Ensembling Judgments for Automatic Relevance Assessmentcited by 6
  36. 2025Time To Impeach LLM-as-a-Judge: Programs are the Future of Evaluationcited by 5
  37. 2023Mitigating Source Bias for Fairer Weak Supervisioncited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 10.

10 papers.

  1. 2026Judging the Judges: A Systematic Evaluation of Bias Mitigation Strategies in LLM-as-a-Judge Pipelinescited by 7
  2. 2026ParetoBandit: Budget-Paced Adaptive Routing for Non-Stationary LLM Servingcited by 3
  3. 2026CoEval: Ranking Language Models for Custom Tasks Without Labeled Data or Trustworthy Benchmarkscited by 2
  4. 2026Are Verifier Errors Independent Within a GRPO Group? Evidence from Qwen2.5 Rolloutscited by 1
  5. 2026LLM-FACETS: A Privacy-Preserving Framework for Evaluating LLM Transparency and Accountabilitycited by 0
  6. 2026Codifying the Judge: Scalable Evaluation via Program Distillationcited by 0
  7. 2026Messier: A High-Resolution Corpus for Cross-Benchmark Agent Evaluationcited by 0
  8. 2026Conditional Evaluation of Language Models with Cheap Auxiliary Signalscited by 0
  9. 2026A Closed-Form Estimator and Diagnostic Battery for Anchor-Judge Error Correlation, Under a Single-Common-Factor Modelcited by 0
  10. 2026RAIM: Robust Aggregation of Inexpensive Models for Hallucination Detectioncited by 0