mlsanityData to 14 Oct 2026

Scaling Model-Generated Distillation Data Can Make Latent Teacher Traits More Recoverable

arXiv 2608.26958 ↗First posted 27 Aug 2026cs.LG cs.CLcited by 2

On the map: Synthetic data and self-improvement for LLMs → LLM synthetic data generation

also in 1 more
Abstract (arXiv)

Scaling model-generated data is usually viewed as improving distillation: more examples should increase coverage, reduce noise, and produce stronger students. We show a second effect: larger datasets can make subtle teacher-specific signals easier to detect in the trained student, even when examples are off-task and never mention the trait. In a controlled setup inspired by subliminal learning, a teacher induced to express a target trait generates restricted off-task data, such as number-only completions. Students trained on different amounts of independent off-task data are evaluated in a separate domain, with matched no-trait controls isolating target-specific transfer. Our main finding is that larger independent datasets make the teacher's induced trait stand out more clearly in the student's later behavior. Other plausible traits may also strengthen with scale, but the target usually grows more. When the small-scale student already favors the target, scaling mainly amplifies that behavior; when it favors a related or salient alternative, more data can shift behavior toward the intended trait. Analyses of learned LoRA updates show a parallel trend. These effects appear across model families, trait types, multi-trait settings, and cross-model transfer. Our results suggest that scaling generated distillation data should be paired with trait-aware curation and evaluation, even when the data appears off-task or benign.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 29 of its 49 references that are in the map. First posted on arXiv 27 Aug 2026.

Builds on

29 of 49 references.

References · 29

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  3. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  4. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  5. 2020Extracting Training Data from Large Language Modelscited by 2,189
  6. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  7. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  8. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  9. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  10. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 924
  11. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  12. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
17 more
  1. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  2. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  3. 2023A Watermark for Large Language Modelscited by 725
  4. 2016Stealing Machine Learning Models via Prediction APIscited by 619
  5. 2023Zephyr: Direct Distillation of LM Alignmentcited by 506
  6. 2013Do Deep Nets Really Need to be Deep?cited by 474
  7. 2023Self-Consuming Generative Models Go MADcited by 250
  8. 2024A Note on Shumailov et al. (2024): `AI Models Collapse When Trained on Recursively Generated Data'cited by 115
  9. 2024LLMmap: Fingerprinting For Large Language Modelscited by 58
  10. 2025Towards Understanding Subliminal Learning: When and How Hidden Biases Transfercited by 34
  11. 2023Students Parrot Their Teachers: Membership Inference on Model Distillationcited by 33
  12. 2023From Text to Source: Results in Detecting Large Language Model-Generated Contentcited by 26
  13. 2026Subliminal Effects in Your Data: A General Mechanism via Log-Linearitycited by 19
  14. 2025Dataset distillation for memorized data: Soft labels can leak held-out teacher knowledgecited by 10
  15. 2026Sustained Gradient Alignment Mediates Subliminal Learning in a Multi-Step Setting: Evidence from MNIST Auxiliary Logit Distillation Experimentcited by 7
  16. 2025Seed-Induced Uniqueness in Transformer Models: Subspace Alignment Governs Subliminal Transfercited by 5
  17. 2026Subliminal Transfer of Unsafe Behaviors in AI Agent Distillationcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Post-Training Leaves Behavioral Shadows on Unrelated Decisionscited by 0
  2. 2026Why Subliminal Learning Needs So Much Data: A Noisy Inverse View through Steering Vector Recoverycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Defence against the dark arts of distillation: Unsupervised discovery of subliminal learning
  2. 2026Cross-Trait Transfer in Subliminal Learning
  3. 2024Why does Knowledge Distillation Work? Rethink its Attention and Fidelity Mechanism
  4. 2026From Gradients to Capabilities: Understanding Multi-Teacher On-Policy Distillation
  5. 2026Knowledge Distillation under Teacher Misspecification: An Order-Parameter Analysis of the Gap between Teacher Mimicry and Task Performance
  6. 2026Subliminal Learning as Trait-Direction Drift: A Mechanism and Targeted Control under SFT Distillation
  7. 2026A Controlled Study of Feature-Based Knowledge Distillation Across Student Designs
  8. 2023Comparative Knowledge Distillation