mlsanityData to 14 Oct 2026

Probably Approximately Correct Labels

arXiv 2506.10908 ↗First posted 12 Jun 2025stat.ML cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

Obtaining high-quality labeled datasets is often costly, requiring either human annotation or expensive experiments. In theory, powerful pre-trained AI models provide an opportunity to automatically label datasets and save costs. Unfortunately, these models come with no guarantees on their accuracy, making wholesale replacement of manual labeling impractical. In this work, we propose a method for leveraging pre-trained AI models to curate cost-effective and high-quality datasets. In particular, our approach results in probably approximately correct labels: with high probability, the overall labeling error is small. Our method is nonasymptotically valid under minimal assumptions on the dataset or the AI model being studied, and thus enables rigorous yet efficient dataset curation using modern AI models. We demonstrate the benefits of the methodology through text annotation with large language models, image labeling with pre-trained vision models, and protein folding analysis with AlphaFold.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 32 of its 54 references that are in the map. First posted on arXiv 12 Jun 2025.

Builds on

32 of 54 references.

References · 32

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,311
  4. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
  5. 2023Whose Opinions Do Language Models Reflect?cited by 773
  6. 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 752
  7. 2019Confident Learning: Estimating Uncertainty in Dataset Labelscited by 413
  8. 2019Label Propagation for Deep Semi-supervised Learningcited by 289
  9. 2021Distribution-Free, Risk-Controlling Prediction Setscited by 288
  10. 2016Data Programming: Creating Large Training Sets, Quicklycited by 246
  11. 2023Prediction-Powered Inferencecited by 220
  12. 2022WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creationcited by 202
20 more
  1. 2020Estimating means of bounded random variables by bettingcited by 169
  2. 2017Cost-Effective Active Learning for Deep Image Classificationcited by 166
  3. 2023PPI++: Efficient Prediction-Powered Inferencecited by 117
  4. 2023CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotationcited by 94
  5. 2020We Can Detect Your Bias: Predicting the Political Ideology of News Articlescited by 92
  6. 2023Cross-Prediction-Powered Inferencecited by 58
  7. 2024Active Statistical Inferencecited by 50
  8. 2024Can Unconfident LLM Annotations Be Used for Confident Conclusions?cited by 49
  9. 2020Detecting Stance in Media on Global Warmingcited by 48
  10. 2021Active label cleaning for improved dataset quality under resource constraintscited by 44
  11. 2021Tight Concentrations and Confidence Sequences from the Regret of Universal Portfoliocited by 43
  12. 2023Using Imperfect Surrogates for Downstream Inference: Design-based Supervised Learning for Social Science Applications of Large Language Modelscited by 39
  13. 2021Misinfo Reaction Frames: Reasoning about Readers' Reactions to News Headlinescited by 32
  14. 2015Active Learning from Weak and Strong Labelerscited by 22
  15. 2021SynthBio: A Case Study in Human-AI Collaborative Curation of Text Datasetscited by 18
  16. 2022How many labelers do you have? A closer look at gold-standard labelscited by 8
  17. 2025Accelerating Unbiased LLM Evaluation via Synthetic Feedbackcited by 7
  18. 2021RiLACS: Risk-Limiting Audits via Confidence Sequencescited by 6
  19. 2023Risk-limiting Financial Audits via Weighted Sampling without Replacementcited by 6
  20. 2020MCAL: Minimum Cost Human-Machine Active Labelingcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 5.

8 papers.

  1. 2025Model-agnostic Selective Labeling with Provable Statistical Guaranteescited by 3
  2. 2025Do We Really Even Need Data? A Modern Look at Drawing Inference with Predicted Datacited by 3
  3. 2026Anytime Safe PAC Efficient Reasoningcited by 1
  4. 2026HyPAC: Cost-Efficient LLMs-Human Hybrid Annotation with PAC Error Guaranteescited by 1
  5. 2026Coverage Guarantees for Pseudo-Calibrated Conformal Prediction under Distribution Shiftcited by 1
  6. 2025The Good, the Bad, and the Sampled: a No-Regret Approach to Safe Online Classificationcited by 0
  7. 2026Uncertainty Quantification for LLM-based Code Generationcited by 0
  8. 2026Robust Conformalized Selection with Noisy Responsescited by 0