# Design-Based Supervised Learning with Noisy Human Labels

arXiv: https://arxiv.org/abs/2607.15455 · HTML: https://mlsanity.com/paper/2607.15455/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 25 references that are in the map.
- First posted on arXiv 16 Jul 2026.

## Builds on (most cited first, 8 of 8)

- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [Prediction-Powered Inference](https://mlsanity.com/paper/2301.09633/) (2023)
- [The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation](https://mlsanity.com/paper/2211.02570/) (2022)
- [LLM-Assisted Content Analysis: Using Large Language Models to Support Deductive Coding](https://mlsanity.com/paper/2306.14924/) (2023)
- [Using Imperfect Surrogates for Downstream Inference: Design-based Supervised Learning for Social Science Applications of Large Language Models](https://mlsanity.com/paper/2306.04746/) (2023)
- [Investigating Reasons for Disagreement in Natural Language Inference](https://mlsanity.com/paper/2209.03392/) (2022)
- [Benchmarking Debiasing Methods for LLM-based Parameter Estimates](https://mlsanity.com/paper/2506.09627/) (2025)
- [From Ground Truth to Measurement: A Statistical Framework for Human Labeling](https://mlsanity.com/paper/2604.07591/) (2026)

## Built on by (most cited first, 1 of 1)

- [Estimating Uncoded Crash Factors with Tabular Foundation and System One Models: Kumo Tabular and Jev](https://mlsanity.com/paper/2610.10321/) (2026)

## Abstract (arXiv)

Researchers increasingly use automated classifiers to label unstructured data for statistical analysis. Existing rectification methods can correct errors in these automated labels using a probability-sampled audit set, but they usually treat the audit labels as correct. In practice, human audit labels are often noisy, and only some audited items are reviewed by an expert or adjudicator. We propose Partially Adjudicated Design-Based Supervised Learning (PA-DSL), a method for this setting. It uses adjudicated cases to correct noisy human labels and then uses the corrected audit information to debias analyses based on the full set of automated labels. The estimator is valid for a broad class of downstream analyses when the audit and adjudication probabilities are known. In synthetic and Wikipedia Detox semi-synthetic experiments, PA-DSL maintains nominal coverage and reduces RMSE by 10-17% relative to using only adjudicated labels when noisy human labels contain recoverable signal.