mlsanityData to 14 Oct 2026

Shared Autoregressive Context Can Distort Relationships in Synthetic Data

arXiv 2609.32546 ↗First posted 26 Sep 2026cs.LG cs.CLcited by 0

On the map: Synthetic data and self-improvement for LLMs → Model collapse from synthetic data also fits: LLM synthetic data generation

Abstract (arXiv)

Large language models can generate several records within one autoregressive completion, making earlier answers available as context for later records. This paper shows that such shared-completion batching can distort relationships among variables in the resulting synthetic data, using controlled tests on synthetic survey respondents. In a matched experiment on 2,000 European Social Survey profiles, generating ten rather than one respondent per request increases mean absolute error in within-country correlations by 48-58% for Qwen3.8-27B and 114-127% for Llama-3.3-70B-Instruct across three seeds, holding profiles, examples, questions and decoding parameters fixed. The distortion primarily reflects exaggerated relationship strength, while retaining substantial agreement with the human ordering of correlations. Controlled interventions establish answer history as a causal channel: re-pairing the same preceding values, with profiles and marginal distributions fixed, changes correlations among subsequently generated responses. Hiding preceding answers reduces correlation error in the tested settings but worsens marginal accuracy. Exploratory corrections across social-attitude, health and economic data likewise show that lower correlation error can coexist with worse marginal distributions and regression estimates. Request construction is therefore part of the data-generating process, and synthetic-data validity must be evaluated against the analyses the generated data are intended to support.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Model collapse from synthetic data, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 7 of its 39 references that are in the map. First posted on arXiv 26 Sep 2026.

Builds on

7 of 39 references.

References · 7

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  2. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,386
  3. 2026Plausible but Not Valid: A Psychometric Audit of LLMs as Synthetic Survey Respondentscited by 8
  4. 2026Beyond the Mean: Three-Axis Fidelity for Aligning LLM-Based Survey Simulators from Small Pilot Datacited by 3
  5. 2026Stochastic Parrots or Singing in Harmony? Testing Five Leading LLMs for their Ability to Replicate a Human Survey with Synthetic Datacited by 2
  6. 2026Tastes without distinction: silicon samples and the synthetic construction of tastescited by 2
  7. 2026The conditional superiority of fast silicon samplingcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Valid Inference with Imperfect Synthetic Data
  2. 2025Emulating Public Opinion: A Proof-of-Concept of AI-Generated Synthetic Survey Responses for the Chilean Case
  3. 2026Synthetic Data in Marketing Research: How to Evaluate and When to Trust
  4. 2026On the Credibility of Evaluating LLMs using Survey Questions
  5. 2025AlignSurvey: A Comprehensive Benchmark for Human Preferences Alignment in Social Surveys
  6. 2026The Fairness Collapse Phenomenon: Bias Amplification in Language Models Trained on Synthetic Data
  7. 2026The Limits of Simulated Societies: How Post-Training and Survey Fine-Tuning Erase Cross-Cultural Variance
  8. 2026Measurement Validity in LLM Cultural Alignment