mlsanityData to 7 Oct 2026

Multi-Expert Conformal Risk Control for Pairwise LLM Judging in Open-Ended Dialogue

arXiv 2608.26529 ↗First posted 27 Aug 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

In this paper, we explore multi-expert Conformal Risk Control (CRC) algorithms for pairwise LLM-as-a-Judge evaluation in open-ended dialogue. Our core insight is that multi-expert aggregation offers a complementary remedy to CRC: whereas CRC controls risk at the decision threshold through abstention, aggregation sanitizes the scoring function at its source. Guided by this, we first design two multi-expert CRC methods: Score Averaging and Decision Voting, which aggregate at the score and decision levels, respectively. While both strategies outperform single-expert methods on homogeneous expert panels, on heterogeneous LLM judges they remain risk-valid but recover only limited coverage, because a uniform threshold cannot match the experts' distinct scoring scales. To resolve this issue, we further propose Marginal-Calibrated Conformal Consensus (MC3): it captures distinct per-expert scales via initial threshold ratios, while jointly tuning a unified decision function Ct(x) applied identically in both calibration and test, thereby preserving exchangeability. To evaluate our framework, we construct Panel, a 1,800-pair human pairwise-preference benchmark for open-ended dialogue. It is built on responses generated by four open-weight LLMs over dialogue contexts from three domains (ESConv, MSC, DREAM), with full logit access. In experiments, we find that both Score Averaging and Decision Voting substantially improve accuracy and acceptance rate on homogeneous panels. Notably, MC3 extends these gains to heterogeneous panels by accommodating distinct per-expert scoring scales across all three datasets.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 27 of its 36 references that are in the map. First posted on arXiv 27 Aug 2026.

Builds on

27 of 36 references.

References · 27

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  4. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
  5. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  6. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
  7. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 301
  8. 2021Towards Emotional Support Dialog Systemscited by 294
  9. 2021Beyond Goldfish Memory: Long-Term Open-Domain Conversationcited by 266
  10. 2025Scalable Best-of-N Selection for Large Language Models via Self-Certaintycited by 182
  11. 2019DREAM: A Challenge Dataset and Models for Dialogue-Based Reading Comprehensioncited by 156
  12. 2024Large Language Models are Inconsistent and Biased Evaluatorscited by 146
15 more
  1. 2023Conformal Prediction with Large Language Models for Multi-Choice Question Answeringcited by 131
  2. 2024Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreementcited by 99
  3. 2024API Is Enough: Conformal Prediction for Large Language Models Without Logit-Accesscited by 68
  4. 2023Adaptation with Self-Evaluation to Improve Selective Prediction in LLMscited by 49
  5. 2024Limits to scalable evaluation at the frontier: LLM as Judge won't beat twice the datacited by 46
  6. 2024Auto-Arena: Automating LLM Evaluations with Agent Peer Battles and Committee Discussionscited by 40
  7. 2024Style Outweighs Substance: Failure Modes of LLM Judges in Alignment Benchmarkingcited by 35
  8. 2024Language Model Council: Democratically Benchmarking Foundation Models on Highly Subjective Taskscited by 17
  9. 2025Chain of Strategy Optimization Makes Large Language Models Better Emotional Supportercited by 17
  10. 2026HEART: A Unified Benchmark for Assessing Humans and LLMs in Emotional Support Dialoguecited by 12
  11. 2026SCOPE: Selective Conformal Optimized Pairwise LLM Judgingcited by 12
  12. 2026Who can we trust? LLM-as-a-jury for Comparative Assessmentcited by 7
  13. 2024CausalScore: An Automatic Reference-Free Metric for Assessing Response Relevance in Open-Domain Dialogue Systemscited by 5
  14. 2024EmPO: Emotion Grounding for Empathetic Response Generation through Preference Optimizationcited by 3
  15. 2025Modeling the One-to-Many Property in Open-Domain Dialogue with LLMscited by 3

Built on by

Not cited yet.