# Fine-Tune, Then Rectify

arXiv: https://arxiv.org/abs/2511.19486 · HTML: https://mlsanity.com/paper/2511.19486/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Nov 2025.

## Built on by (most cited first, 7 of 7)

- [Calibeating Prediction-Powered Inference](https://mlsanity.com/paper/2604.21260/) (2026)
- [Generative Augmented Inference of LLM-generated Data for Market Research: Theory and Empirical Evidence](https://mlsanity.com/paper/2604.14575/) (2026)
- [Allocating Human Oversight in AI-Enabled Analytics](https://mlsanity.com/paper/2604.12497/) (2026)
- [When Can You Trust Your Synthetic Users? Diagnostics and Corrections for LLM Consumer Panels](https://mlsanity.com/paper/2609.13148/) (2026)
- [AI-Generated Measurements for Identification and Inference with Missing Data: A Weak Shadow Variable Approach](https://mlsanity.com/paper/2602.16061/) (2026)
- [Rectification Difficulty and Optimal Sample Allocation in LLM-Augmented Surveys](https://mlsanity.com/paper/2604.17267/) (2026)
- [The Scaling Paradox in Human-AI Collaboration](https://mlsanity.com/paper/2608.00818/) (2026)

## Abstract (arXiv)

Driven by recent advances in artificial intelligence, a growing literature has demonstrated the potential of using large language models (LLMs) as scalable surrogates to generate human-like responses. Two common approaches to improve the performance of LLMs include: fine-tuning, which aligns the LLM more closely with human responses, and rectification, which corrects biases in LLM outputs. In this paper, we develop a two-stage framework that combines fine-tuning and rectification, and optimally allocates limited labeled samples across the two stages. A key insight is that the conventional fine-tuning objective of minimizing mean squared prediction error is generally not aligned with the downstream rectification stage. For mean estimation, we propose to minimize the variance of the prediction errors; for general M-estimation, we propose to minimize a scalarized variance metric as the fine-tuning objective. Building on this insight, we leverage the scaling law of fine-tuning to optimally allocate the limited labeled human data between the fine-tuning and rectification stages. Our empirical analysis validates the fine-tuning scaling law and confirms that our proposed optimal allocation rule reliably identifies the optimal sample allocation. We demonstrate substantial efficiency gains in estimation and inference performance relative to fine-tuning or rectification alone, or to employing the conventional mean squared error objective within the fine-tuning then rectification framework. Such efficiency gains translate to significant cost savings for making reliable decisions.