mlsanityData to 7 Oct 2026

HyPAC: Cost-Efficient LLMs-Human Hybrid Annotation with PAC Error Guarantees

arXiv 2602.02550 ↗First posted 30 Jan 2026cs.LG cs.AIcited by 1

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Data annotation often involves multiple sources with different cost-quality trade-offs, such as fast large language models (LLMs), slow reasoning models, and human experts. In this work, we study the problem of routing inputs to the most cost-efficient annotation source while controlling the labeling error on test instances. We propose HyPAC, a method that adaptively labels inputs to the most cost-efficient annotation source while providing distribution-free guarantees on annotation error. HyPAC calibrates two decision thresholds using importance sampling and upper confidence bounds, partitioning inputs into three regions based on uncertainty and routing each to the appropriate annotation source. We prove that HyPAC achieves the minimum expected cost with a probably approximately correct (PAC) guarantee on the annotation error, free of data distribution and pre-trained models. Experiments on common benchmarks demonstrate the effectiveness of our method, reducing the annotation cost by 78.51% while tightly controlling the annotation error.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 36 of its 46 references that are in the map. First posted on arXiv 30 Jan 2026.

Builds on

36 of 46 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters6 Aug
  2. 2024Uncertainty of Thoughts: Uncertainty-Aware Planning Enhances Information Seeking in Large Language Models5 Feb

Other references · 34

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  5. 2017On Calibration of Modern Neural Networkscited by 4,860
  6. 2024Qwen2.5 Technical Reportcited by 4,713
  7. 2023Let's Verify Step by Stepcited by 3,598
  8. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  9. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  10. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  11. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  12. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
22 more
  1. 2022Conformal Risk Controlcited by 290
  2. 2025Chain of Draft: Thinking Faster by Writing Lesscited by 290
  3. 2021Distribution-Free, Risk-Controlling Prediction Setscited by 278
  4. 2021Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Controlcited by 256
  5. 2022Is GPT-3 a Good Data Annotator?cited by 208
  6. 2025ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoningcited by 186
  7. 2024Are We Done with MMLU?cited by 178
  8. 2023AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotatorscited by 171
  9. 2020Estimating means of bounded random variables by bettingcited by 164
  10. 2023Conformal Language Modelingcited by 163
  11. 2023Large Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoningcited by 140
  12. 2024Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreementcited by 99
  13. 2023CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotationcited by 93
  14. 2024On Verbalized Confidence Scores for LLMscited by 83
  15. 2024A Unified Approach to Routing and Cascading for LLMscited by 71
  16. 2024Can Unconfident LLM Annotations Be Used for Confident Conclusions?cited by 44
  17. 2024Localized Adaptive Risk Controlcited by 16
  18. 2024Automatically Adaptive Conformal Risk Controlcited by 16
  19. 2025A Case Study of Scalable Content Annotation Using Multi-LLM Consensus and Human Reviewcited by 7
  20. 2025Probably Approximately Correct Labelscited by 7
  21. 2025CrowdAgent: Multi-Agent Managed Multi-Source Annotation Systemcited by 5
  22. 2024Human-LLM Hybrid Text Answer Aggregation for Crowd Annotationscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026QUORUM: QUality-Optimized Routing Using Multiple annotatorscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025From LLM-anation to LLM-orchestrator: Coordinating Small Models for Data Labeling
  2. 2025Evaluating LLMs Without Oracle Feedback: Agentic Annotation Evaluation Through Unsupervised Consistency Signals
  3. 2025AI-Powered Annotation Pipelines for Stabilizing Large Language Models: A Human-AI Synergy Approach
  4. 2025How Humans Help LLMs: Assessing and Incentivizing Human Preference Annotators
  5. 2024MEGAnno+: A Human-LLM Collaborative Annotation System
  6. 2025ACT as Human: Multimodal Large Language Model Data Annotation with Critical Thinking
  7. 2026Auditing Automated Evaluation, Error Propagation, and Runtime Mitigation in Tool-Using Language Agents
  8. 2026More Human, More Efficient: Aligning Annotations with Quantized SLMs