mlsanityData to 7 Oct 2026

SCOPE: Selective Conformal Optimized Pairwise LLM Judging

arXiv 2602.13110 ↗First posted 13 Feb 2026cs.CL cs.AIcited by 12

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) are increasingly used as scalable judges in pairwise evaluation, but they remain prone to miscalibration and biases. We propose textscScope (Selective Conformal Optimized Pairwise Evaluation), a framework that calibrates an acceptance threshold so that, under exchangeability, the error rate among non-abstained judgments is at most a user-specified level α. To supply textscScope with a bias-neutral uncertainty signal, we introduce Bidirectional Preference Entropy (BPE), which queries the judge under both response positions and converts the order-averaged preference probability into an entropy-based score. Across various pairwise judging benchmarks, BPE outperforms standard confidence proxies in calibration and discrimination, while textscScope consistently satisfies the target risk bound (empirical FDR ≈ 0.097--0.099 at α=0.10) and retains substantial coverage. Compared to vanilla baselines, textscScope accepts up to 2.4× more judgments under the same risk constraint, demonstrating that BPE enables reliable and high-coverage LLM-based evaluation.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 32 of its 44 references that are in the map. First posted on arXiv 13 Feb 2026.

Builds on

32 of 44 references.

References · 32

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  5. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  6. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  7. 2017Selective Classification for Deep Neural Networkscited by 799
  8. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  9. 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 777
  10. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  11. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
  12. 2019Conformal Prediction Under Covariate Shiftcited by 511
20 more
  1. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  2. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 294
  3. 2022Conformal Risk Controlcited by 290
  4. 2021Distribution-Free, Risk-Controlling Prediction Setscited by 278
  5. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 252
  6. 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 249
  7. 2024Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreementcited by 99
  8. 2024Calibrating Large Language Models with Sample Consistencycited by 72
  9. 2024ConU: Conformal Uncertainty in Large Language Models with Correctness Coverage Guaranteescited by 62
  10. 2024Eliminating Position Bias of Language Models: A Mechanistic Approachcited by 61
  11. 2024Conformal Alignment: Knowing When to Trust Foundation Models with Guaranteescited by 50
  12. 2023Adaptation with Self-Evaluation to Improve Selective Prediction in LLMscited by 49
  13. 2024Reference-Guided Verdict: LLMs-as-Judges in Automatic Evaluation of Free-Form QAcited by 25
  14. 2025Pairwise or Pointwise? Evaluating Feedback Protocols for Bias in LLM-Based Evaluationcited by 23
  15. 2025An Empirical Analysis of Uncertainty in Large Language Model Evaluationscited by 20
  16. 2025COIN: Uncertainty-Guarding Selective Question Answering for Foundation Models with Provable Risk Guaranteescited by 20
  17. 2025TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Themcited by 20
  18. 2024Mitigating Hallucinations in Large Language Models via Self-Refinement-Enhanced Knowledge Retrievalcited by 16
  19. 2025Conformal Inference under High-Dimensional Covariate Shifts via Likelihood-Ratio Regularizationcited by 11
  20. 2025CLEV: LLM-Based Evaluation Through Lightweight Efficient Voting for Free-Form Question-Answeringcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 12.

12 papers.

  1. 2026A Finite-Calibration Regime Map for LLM Judge Panelscited by 2
  2. 2026CARE: A Conformal Safety Layer for Medical Summarizationcited by 1
  3. 2026Judge, Retrieve, or Abstain: Uncertainty-Guarded LLM Judging with Provable Risk Guaranteescited by 1
  4. 2026Share the Judge, Learn the Deferral: Where Specialization Helps LLM Evaluationcited by 0
  5. 2026Uncertainty quantification for trustworthy deep learning: Methods and measurescited by 0
  6. 2026Localize-Then-Decide Guarantees for LLM Judgmentscited by 0
  7. 2026Multi-Expert Conformal Risk Control for Pairwise LLM Judging in Open-Ended Dialoguecited by 0
  8. 2026Aggregate Disambiguation Systemscited by 0
  9. 2026Solving versus Verifying: Catching Contradictions in Tax Reasoning Systemscited by 0
  10. 2026Decomposing LLM-Judge Uncertainty to Target Expert Labelscited by 0
  11. 2026Backbone-Adaptive Evidence Routing for Robust Pairwise LLM Judgingcited by 0
  12. 2026Certified Selective Automation of LLM Agent Evaluationcited by 0