# Benchmarking Debiasing Methods for LLM-based Parameter Estimates

arXiv: https://arxiv.org/abs/2506.09627 · HTML: https://mlsanity.com/paper/2506.09627/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 33 references that are in the map.
- First posted on arXiv 11 Jun 2025.

## Builds on (most cited first, 9 of 9)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting](https://mlsanity.com/paper/1901.09451/) (2019)
- [Prediction-Powered Inference with Imputed Covariates and Nonuniform Sampling](https://mlsanity.com/paper/2501.18577/) (2025)
- [Conceptualizing Treatment Leakage in Text-based Causal Inference](https://mlsanity.com/paper/2205.00465/) (2022)
- [Computational Politeness in Natural Language Processing: A Survey](https://mlsanity.com/paper/2407.12814/) (2024)
- [Can Large Language Models (or Humans) Disentangle Text?](https://mlsanity.com/paper/2403.16584/) (2024)
- [Yesterday's News: Benchmarking Multi-Dimensional Out-of-Distribution Generalization of Misinformation Detection Models](https://mlsanity.com/paper/2410.18122/) (2024)

## Built on by (most cited first, 6 of 6)

- [Debiasing Machine Learning Predictions for Causal Inference Without Additional Ground Truth Data: "One Map, Many Trials" in Satellite-Driven Poverty Analysis](https://mlsanity.com/paper/2508.01341/) (2025)
- [What is a protest anyway? Codebook conceptualization is still a first-order concern in LLM-era classification](https://mlsanity.com/paper/2510.03541/) (2025)
- [Platonic Representations for Poverty Mapping: Unified Vision-Language Codes or Agent-Induced Novelty?](https://mlsanity.com/paper/2508.01109/) (2025)
- [Design-Based Supervised Learning with Noisy Human Labels](https://mlsanity.com/paper/2607.15455/) (2026)
- [Beyond Interaction Effects: Two Logics for Studying Population Inequalities](https://mlsanity.com/paper/2601.04223/) (2025)
- [How to Run Statistics over LLM Judges and Trust the Results: Calibrated Inference for Small-Sample AI Evaluation with evalstats](https://mlsanity.com/paper/2609.35815/) (2026)

## Abstract (arXiv)

Large language models (LLMs) offer an inexpensive yet powerful way to annotate text, but are often inconsistent when compared with experts. These errors can bias downstream estimates of population parameters such as regression coefficients and causal effects. To mitigate this bias, researchers have developed debiasing methods such as Design-based Supervised Learning (DSL) and Prediction-Powered Inference (PPI), which promise valid estimation by combining LLM annotations with a limited number of expensive expert annotations. Although these methods produce consistent estimates under theoretical assumptions, it is unknown how they compare in finite samples of sizes encountered in applied research. We make two contributions. First, we study how each methods performance scales with the number of expert annotations, highlighting regimes where LLM bias or limited expert labels significantly affect results. Second, we compare DSL and PPI across a range of tasks, finding that although both achieve low bias with large datasets, DSL often outperforms PPI on bias reduction and empirical efficiency, but its performance is less consistent across datasets. Our findings indicate that there is a bias-variance tradeoff at the level of debiasing methods, calling for more research on developing metrics for quantifying their efficiency in finite samples.