# Self-Study Reconsidered: The Hidden Fragility of Learning from Self-Generated QA

arXiv: https://arxiv.org/abs/2606.32002 · HTML: https://mlsanity.com/paper/2606.32002/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Synthetic reasoning data and bootstrapping, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 35 of its 51 references that are in the map.
- First posted on arXiv 30 Jun 2026.

## Placed in

- Synthetic data and self-improvement for LLMs / [Synthetic reasoning data and bootstrapping](https://mlsanity.com/synth/synthetic-reasoning-data-and-bootstrapping/)

## Builds on (most cited first, 30 of 35)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://mlsanity.com/paper/2302.12173/) (2023)
- [Ignore Previous Prompt: Attack Techniques For Language Models](https://mlsanity.com/paper/2211.09527/) (2022)
- [Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting](https://mlsanity.com/paper/2310.11324/) (2023)
- [InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents](https://mlsanity.com/paper/2403.02691/) (2024)
- [The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions](https://mlsanity.com/paper/2404.13208/) (2024)
- [A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers](https://mlsanity.com/paper/2105.03011/) (2021)
- [Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor](https://mlsanity.com/paper/2212.09689/) (2022)
- [StruQ: Defending Against Prompt Injection with Structured Queries](https://mlsanity.com/paper/2402.06363/) (2024)
- [Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing](https://mlsanity.com/paper/2406.08464/) (2024)
- [Poisoning Web-Scale Training Datasets is Practical](https://mlsanity.com/paper/2302.10149/) (2023)
- [Physics of Language Models: Part 3.1, Knowledge Storage and Extraction](https://mlsanity.com/paper/2309.14316/) (2023)
- [PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models](https://mlsanity.com/paper/2402.07867/) (2024)
- [Defending Against Indirect Prompt Injection Attacks With Spotlighting](https://mlsanity.com/paper/2403.14720/) (2024)
- [ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2311.09476/) (2023)
- [Promptagator: Few-shot Dense Retrieval From 8 Examples](https://mlsanity.com/paper/2209.11755/) (2022)
- [Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injection](https://mlsanity.com/paper/2307.16888/) (2023)
- [On the Exploitability of Instruction Tuning](https://mlsanity.com/paper/2306.17194/) (2023)
- [PromptArmor: Simple yet Effective Prompt Injection Defenses](https://mlsanity.com/paper/2507.15219/) (2025)
- [Best Practices and Lessons Learned on Synthetic Data](https://mlsanity.com/paper/2404.07503/) (2024)
- [Self-Adapting Language Models](https://mlsanity.com/paper/2506.10943/) (2025)
- [InPars-v2: Large Language Models as Efficient Dataset Generators for Information Retrieval](https://mlsanity.com/paper/2301.01820/) (2023)
- [Synthetic continued pretraining](https://mlsanity.com/paper/2409.07431/) (2024)
- [Parametric Retrieval Augmented Generation](https://mlsanity.com/paper/2501.15915/) (2025)
- [LongHealth: A Question Answering Benchmark with Long Clinical Documents](https://mlsanity.com/paper/2401.14490/) (2024)
- [Genie: Achieving Human Parity in Content-Grounded Datasets Generation](https://mlsanity.com/paper/2401.14367/) (2024)

## Abstract (arXiv)

Language models are increasingly taught from synthetic question--answer (QA) supervision: a model generates questions about a document, answers them from the same text, and the resulting pairs are used to fine-tune, distill, or compress knowledge into another model. We show that this generation step is not neutral preprocessing. It is an implicit policy that both selects which evidence becomes training signal and decides how that evidence is answered, and it is fragile at both stages. When choosing what to ask, generators do not scan a document uniformly. Coverage saturates early and concentrates on salient spans, diverse prompts converge on the same regions, and what looks question-worthy is driven by local presentation. As a result, salient artifacts such as poorly cleaned markup can hijack question generation across model families and scales. When answering, the model that produces the supervision tends to obey instruction-like passages embedded in the text. This compliance depends on the intent and surface form of the passage rather than its strictness, and is worst under task conflict, where larger models comply more often. These failure modes arise from choices made during QA generation, so they can be reduced without changing the training loop. Tying each question to a fixed target reduces biased selection, and filtering instruction-like spans before answering lowers mean injection compliance from 88% to 13% in our evaluation while retaining nearly all clean text.