# Valid Inference with Synthetic Data via Task Exchangeability

arXiv: https://arxiv.org/abs/2606.13629 · HTML: https://mlsanity.com/paper/2606.13629/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 27 of its 39 references that are in the map.
- First posted on arXiv 11 Jun 2026.

## Placed in

- Synthetic data and self-improvement for LLMs / [LLM synthetic data generation](https://mlsanity.com/synth/llm-synthetic-data-generation/)

## Builds on (most cited first, 27 of 27)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [Whose Opinions Do Language Models Reflect?](https://mlsanity.com/paper/2303.17548/) (2023)
- [Out of One, Many: Using Language Models to Simulate Human Samples](https://mlsanity.com/paper/2209.06899/) (2022)
- [Conformal Prediction Under Covariate Shift](https://mlsanity.com/paper/1904.06019/) (2019)
- [Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge](https://mlsanity.com/paper/2410.02736/) (2024)
- [Conformal prediction beyond exchangeability](https://mlsanity.com/paper/2202.13415/) (2022)
- [Prediction-Powered Inference](https://mlsanity.com/paper/2301.09633/) (2023)
- [PPI++: Efficient Prediction-Powered Inference](https://mlsanity.com/paper/2311.01453/) (2023)
- [Localized Conformal Prediction: A Generalized Inference Framework for Conformal Prediction](https://mlsanity.com/paper/2106.08460/) (2021)
- [Theoretical Foundations of Conformal Prediction](https://mlsanity.com/paper/2411.11824/) (2024)
- [Cross-Prediction-Powered Inference](https://mlsanity.com/paper/2309.16598/) (2023)
- [Conformal Prediction Under Feedback Covariate Shift for Biomolecular Design](https://mlsanity.com/paper/2202.03613/) (2022)
- [Predictions as Surrogates: Revisiting Surrogate Outcomes in the Age of AI](https://mlsanity.com/paper/2501.09731/) (2025)
- [Assumption-Lean and Data-Adaptive Post-Prediction Inference](https://mlsanity.com/paper/2311.14220/) (2023)
- [Prediction-Powered Inference with Imputed Covariates and Nonuniform Sampling](https://mlsanity.com/paper/2501.18577/) (2025)
- [How Many Human Survey Respondents is a Large Language Model Worth? An Uncertainty Quantification Perspective](https://mlsanity.com/paper/2502.17773/) (2025)
- [Demystifying Prediction Powered Inference](https://mlsanity.com/paper/2601.20819/) (2026)
- [Multiple-Prediction-Powered Inference](https://mlsanity.com/paper/2603.27414/) (2026)
- [Prediction-Powered Adaptive Shrinkage Estimation](https://mlsanity.com/paper/2502.14166/) (2025)
- [Valid Inference with Imperfect Synthetic Data](https://mlsanity.com/paper/2508.06635/) (2025)
- [The Real Deal Behind the Artificial Appeal: Inferential Utility of Tabular Synthetic Data](https://mlsanity.com/paper/2312.07837/) (2023)
- [GLM Inference with AI-Generated Synthetic Data Using Misspecified Linear Regression](https://mlsanity.com/paper/2503.21968/) (2025)
- [Imputation-Powered Inference](https://mlsanity.com/paper/2509.13778/) (2025)
- [Power Analysis for Prediction-Powered Inference](https://mlsanity.com/paper/2603.16041/) (2026)
- [Prediction-Powered Inference Across Many Tasks for AI Evaluation & Social Science Research](https://mlsanity.com/paper/2605.29249/) (2026)

## Built on by (most cited first, 1 of 1)

- [Learning a Size-Weight Frontier for Synthetic-Augmented Inference](https://mlsanity.com/paper/2608.28576/) (2026)

## Abstract (arXiv)

There is a proliferation of work arguing for the use of synthetic data in scientific research. For example, social scientists are arguing for the use of LLM-generated "silicon samples" in pilot studies; AI evaluations increasingly rely on "LLM-as-a-judge" outputs; and proteomics research is accelerated by generative models that produce synthetic protein structures. These developments raise an intriguing possibility: synthetic data may help researchers ask more questions, run more studies, and accelerate discovery. But they also raise a fundamental concern: synthetic data can be biased, noisy, and misspecified. In this work, we propose statistical principles for using synthetic data in scientific research with provable validity guarantees. The key insight is a new technical condition that we call task exchangeability. Informally, this is a requirement that the researcher can identify historical tasks, for which real data is available, such that their current task of interest is exchangeable with the historical tasks in an appropriate mathematical sense. We develop methods for valid inference under task exchangeability, together with extensions that provide guarantees even beyond exchangeability. We demonstrate the framework on public opinion surveys with silicon samples and AI evaluation with autoraters.