mlsanityData to 14 Oct 2026

Risk-Controlled Selective LLM Answering by Pricing Label-Free Checks

arXiv 2609.37493 ↗First posted 27 Sep 2026cs.LG cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Serving an answer from a large language model requires deciding when to abstain, yet a verifier's ranking accuracy alone does not determine the error rate among served answers. We introduce PriceCheck, which builds a compact family of decision rules from label-free checks such as re-solving a problem. Each check has a price: its agreement rates on correct and incorrect answers and its cost per run. Prices fitted on a small, class-enriched labelled set compose into predictions of a schedule's coverage and cost, guiding which checks to run and when to stop. A calibration test then selects a schedule at a stated selective-risk target. In mathematics, the selected schedules serve 76.1% of answers on average and keep held-out selective risk below 1.5% on all 15 splits. Under the shared testing protocol, PriceCheck serves more answers at that target than reward models, a prompted judge, the generator's confidence and a trained correctness classifier. At matched coverage, it keeps the fewest wrong answers among these scorers. Across 118 diagnostic schedules, price-based coverage predictions have a rank correlation of 0.97 with observed coverage. These results show that choosing how checks are combined and stopped matters alongside how well a verifier ranks answers. Code is available at https://github.com/js-lee-AI/PriceCheck.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 37 of its 52 references that are in the map. First posted on arXiv 27 Sep 2026.

Builds on

37 of 52 references.

References · 37

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  7. 2021Program Synthesis with Large Language Modelscited by 3,637
  8. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  9. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  10. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  11. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  12. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
25 more
  1. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  2. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  3. 2022Solving math word problems with process- and outcome-based feedbackcited by 901
  4. 2017Selective Classification for Deep Neural Networkscited by 852
  5. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 819
  6. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
  7. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 810
  8. 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 792
  9. 2019Conformal Prediction Under Covariate Shiftcited by 519
  10. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480
  11. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoningcited by 415
  12. 2022Large Language Models are Better Reasoners with Self-Verificationcited by 307
  13. 2021Distribution-Free, Risk-Controlling Prediction Setscited by 288
  14. 2021Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Controlcited by 271
  15. 2020Selective Question Answering under Domain Shiftcited by 238
  16. 2023Deductive Verification of Chain-of-Thought Reasoningcited by 124
  17. 2023Let's Sample Step by Step: Adaptive-Consistency for Efficient Reasoning and Coding with LLMscited by 115
  18. 2024Escape Sky-high Cost: Early-stopping Self-Consistency for Multi-step Reasoningcited by 108
  19. 2023GPT-4 Doesn't Know It's Wrong: An Analysis of Iterative Prompting for Reasoning Problemscited by 89
  20. 2023RCOT: Detecting and Rectifying Factual Inconsistency in Reasoning by Reversing Chain-of-Thoughtcited by 47
  21. 2024Unsupervised Evaluation of Code LLMs with Round-Trip Correctnesscited by 23
  22. 2025An Empirical Study of LLM-as-a-Judge: How Design Choices Impact Evaluation Reliabilitycited by 19
  23. 2026DART: Draft-Agreement Routing for Training-Free Adaptive Thinking Budgets in Hybrid Reasoning Modelscited by 8
  24. 2026Answer-Conditioned Chains of Thought Degrade Verifiable-Reasoning Distillation in Large Language Modelscited by 7
  25. 2025Hallucinate Less by Thinking More: Aspect-Based Causal Abstention for Large Language Modelscited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Economic Evaluation of LLMs
  2. 2025Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMs
  3. 2026Conformal Thinking: Risk Control for Reasoning on a Compute Budget
  4. 2026Learning the Cost of Reliable Inference
  5. 2025CompassVerifier: A Unified and Robust Verifier for LLMs Evaluation and Outcome Reward
  6. 2026Adaptive Power Sampling for LLM Reasoning
  7. 2026Interpretable Adaptive Sampling for LLM Test-Time Scaling
  8. 2026The Shadow Price of Reasoning: Economic Perspective on Optimal Budget Allocation for LLMs