mlsanityData to 14 Oct 2026

Towards Understanding Subliminal Learning: When and How Hidden Biases Transfer

arXiv 2509.23886 ↗First posted 28 Sep 2025cs.LG cs.AIcited by 34

On the map: Knowledge distillation → On-policy and self-distillation for LLMs

Abstract (arXiv)

Language models can transfer hidden biases during distillation. For example, a teacher that "likes owls" can make its student "like owls" too, even when the training data consists only of lists of numbers. This surprising phenomenon is called subliminal learning. Subliminal learning can be expected under soft distillation, where the student is trained on the teacher's full next-token distribution. But the fact that this also occurs under hard distillation-where the student only sees sampled tokens-raises a deeper question: when and how does subliminal learning actually occur? We answer this question through controlled experiments and mechanistic analysis. Our results show that subliminal learning does not need (global) token entanglement or logit leakage. Instead, it comes down to a small set of divergence tokens-rare cases where teachers with different biases would predict different tokens. Masking out these tokens mostly removes the hidden bias transfer. Mechanistically, divergence tokens reveal that early layers are critical. Surprisingly, finetuning even a single such early layer is sufficient for subliminal learning. Finally, we find that subliminal learning is fragile. Even small changes, like prompt paraphrasings, are usually sufficient to suppress it.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Placed in On-policy and self-distillation for LLMs, one of 12 directions in Knowledge distillation. Builds on 33 of its 46 references that are in the map. First posted on arXiv 28 Sep 2025.

Builds on

33 of 46 references.

References · 33

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2024Qwen2.5 Technical Reportcited by 4,942
  6. 2017Axiomatic Attribution for Deep Networkscited by 3,384
  7. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  8. 2025Gemma 3 Technical Reportcited by 1,824
  9. 2016Concrete Problems in AI Safetycited by 1,642
  10. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  11. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 886
  12. 2024Phi-4 Technical Reportcited by 711
21 more
  1. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  2. 2018Born Again Neural Networkscited by 546
  3. 2024Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Trainingcited by 529
  4. 2019Quantifying the Carbon Emissions of Machine Learningcited by 521
  5. 2018Poison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networkscited by 519
  6. 2013Do Deep Nets Really Need to be Deep?cited by 474
  7. 2019Hidden Trigger Backdoor Attackscited by 371
  8. 2024Deliberative Alignment: Reasoning Enables Safer Language Modelscited by 308
  9. 2013Direct and Indirect Effectscited by 280
  10. 2019Risks from Learned Optimization in Advanced Machine Learning Systemscited by 258
  11. 2022The Alignment Problem from a Deep Learning Perspectivecited by 258
  12. 2024Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanismscited by 149
  13. 2023Scheming AIs: Will AIs fake alignment during training in order to get power?cited by 97
  14. 2025Subliminal Learning: Language models transmit behavioral traits via hidden signals in datacited by 95
  15. 2025Model Organisms for Emergent Misalignmentcited by 91
  16. 2025Open Problems in Machine Unlearning for AI Safetycited by 51
  17. 2023Closing the Curious Case of Neural Text Degenerationcited by 32
  18. 2025Layer-Aware Representation Filtering: Purifying Finetuning Data to Preserve LLM Safety Alignmentcited by 24
  19. 2021Towards Understanding Adversarial Robustness of Optical Flow Networkscited by 15
  20. 2025Dataset distillation for memorized data: Soft labels can leak held-out teacher knowledgecited by 10
  21. 2025Beyond Black-Box Obfuscation: Mechanistic Analysis and Defense of White-Box Monitorscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 34.

34 papers.

Sort:

By area:

  1. 2026Subliminal Effects in Your Data: A General Mechanism via Log-Linearitycited by 19
  2. 2026Subliminal Steering: Stronger Encoding of Hidden Signalscited by 14
  3. 2026Subliminal Learning Is Steering Vector Distillationcited by 13
  4. 2026Thought Virus: Viral Misalignment via Subliminal Prompting in Multi-Agent Systemscited by 9
  5. 2026From Data to Behavior: Predicting Unintended Model Behaviors Before Trainingcited by 8
  6. 2026You Didn't Have to Say It like That: Subliminal Learning from Faithful Paraphrasescited by 8
  7. 2026Emergent and Subliminal Misalignment Through the Lens of Data-Mediated Transfercited by 8
  8. 2026Learning Through Noise: Why Subliminal Learning Works and When It Failscited by 7
  9. 2026Channel Location Constrains the Auditability of Subliminal Learningcited by 7
  10. 2026Subliminal Learning is a LoRA Artifactcited by 6
  11. 2026Covert Influence Between Language Modelscited by 6
  12. 2026Subliminal Signals in Preference Labelscited by 3
22 more
  1. 2026Subliminal Learning is Non-Semantic Distillationcited by 3
  2. 2026Stored in Optimizer State, Valued by Later Training: A Causal Account of Subliminal Trait Transfercited by 3
  3. 2026The Piggyback Hypothesis of Generalization: Explaining and Mitigating Emergent Misalignmentcited by 2
  4. 2026Covert Trait Propagation Is Representation Alignment: Mechanistic Evidence from Hidden-Channel Distillationcited by 2
  5. 2026Mechanist: AI as a Scientific Instrument for Discovering the Mechanisms of Intelligencecited by 2
  6. 2026Scaling Model-Generated Distillation Data Can Make Latent Teacher Traits More Recoverablecited by 2
  7. 2026Objective Matters: Fine-Tuning Objectives Shape Safety, Robustness, and Persona Driftcited by 1
  8. 2026Inference-Time Consensus for Mitigating Hidden Behaviors from LLM Fine-Tuningcited by 1
  9. 2026Subliminal Learning as Trait-Direction Drift: A Mechanism and Targeted Control under SFT Distillationcited by 1
  10. 2026Hidden Threat in Synthetic Data: Covert Targeted Bias Injection through Benign Textcited by 0
  11. 2026Towards Identifying the Dataset Biases Causing Phantom Transfercited by 0
  12. 2026Subliminal Prompting Beyond Static Geometry: Causal Depth and Multi-Token Confoundscited by 0
  13. 2026Can Data Attribution Filter Out Subliminal Learning? Not Reliablycited by 0
  14. 2026On Mitigation of Subliminal Learning in Large Language Modelscited by 0
  15. 2026Why Ghost Outputs Teach: A Kernel-Based Understanding of Subliminal Learningcited by 0
  16. 2026Slow Decay and Silenced Expression: Iterated Subliminal Trait Transfer in Language-Model Lineagescited by 0
  17. 2026Post-Training Leaves Behavioral Shadows on Unrelated Decisionscited by 0
  18. 2026Cross-Trait Transfer in Subliminal Learningcited by 0
  19. 2026Why Subliminal Learning Needs So Much Data: A Noisy Inverse View through Steering Vector Recoverycited by 0
  20. 2026Distillation for Incrimination and Distillation for Capabilitiescited by 0
  21. 2026Defence against the dark arts of distillation: Unsupervised discovery of subliminal learningcited by 0
  22. 2026Imprinting: Knowledge Distillation Can Shift Student Behaviorcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Owls: Subliminal Learning Can Transfer Learned Capabilities and Backdoors
  2. 2026Quantifying Subliminal Behavioral Transfer Ratios in Language Model Distillation
  3. 2026Verbalizing Subliminal Learning Effects Using Text Optimization
  4. 2026Sustained Gradient Alignment Mediates Subliminal Learning in a Multi-Step Setting: Evidence from MNIST Auxiliary Logit Distillation Experiment
  5. 2026Divergence controls entropy in distillation
  6. 2026Reproducing and Evaluating the Generalizability of Subliminal Learning in Open-Weight Models
  7. 2026Disentangling Self-Distillation: Measuring and Modeling Acquisition and Retention
  8. 2026Anti-Shortcut Distillation via Temporal Negative Knowledge Transfer