# Quantifying Subliminal Behavioral Transfer Ratios in Language Model Distillation

arXiv: https://arxiv.org/abs/2606.11270 · HTML: https://mlsanity.com/paper/2606.11270/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 19 references that are in the map.
- First posted on arXiv 9 Jun 2026.

## Builds on (most cited first, 17 of 17)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Representation Engineering: A Top-Down Approach to AI Transparency](https://mlsanity.com/paper/2310.01405/) (2023)
- [Refusal in Language Models Is Mediated by a Single Direction](https://mlsanity.com/paper/2406.11717/) (2024)
- [JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models](https://mlsanity.com/paper/2404.01318/) (2024)
- [Subliminal Learning: Language models transmit behavioral traits via hidden signals in data](https://mlsanity.com/paper/2507.14805/) (2025)
- [A Comprehensive Survey on Knowledge Distillation](https://mlsanity.com/paper/2503.12067/) (2025)
- [SALT: Steering Activations towards Leakage-free Thinking in Chain of Thought](https://mlsanity.com/paper/2511.07772/) (2025)
- [Beyond Black-Box Obfuscation: Mechanistic Analysis and Defense of White-Box Monitors](https://mlsanity.com/paper/2505.14300/) (2025)
- [Mechanistic Anomaly Detection for "Quirky" Language Models](https://mlsanity.com/paper/2504.08812/) (2025)
- [Playing Devil's Advocate: Off-the-Shelf Persona Vectors Rival Targeted Steering for Sycophancy](https://mlsanity.com/paper/2605.21006/) (2026)
- [Mechanistic origins of catastrophic forgetting: why RL preserves circuits better than SFT?](https://mlsanity.com/paper/2605.28860/) (2026)

## Built on by (most cited first, 3 of 3)

- [Stored in Optimizer State, Valued by Later Training: A Causal Account of Subliminal Trait Transfer](https://mlsanity.com/paper/2608.20442/) (2026)
- [Beyond Owls: Subliminal Learning Can Transfer Learned Capabilities and Backdoors](https://mlsanity.com/paper/2610.10657/) (2026)
- [Imprinting: Knowledge Distillation Can Shift Student Behavior](https://mlsanity.com/paper/2610.18517/) (2026)

## Abstract (arXiv)

Distillation of a language model intended to transfer benign behavior to a student model may also transfer undesirable characteristics, if they are present in the teacher model, a phenomenon known as subliminal learning. While qualitative evidence supports the existence of this effect, its magnitude has not been systematically characterized. This study quantifies subliminal behavioral transfer ratios by steering two teacher models (Llama-2-7B-Chat and Qwen2.5-7B-Instruct) at varying steering strengths and distilling student models using only benign data. Evaluation on 100 JailbreakBench prompts with GPT-4.1, serving as the evaluator, indicates that transfer is robust but exhibits distinct scaling behaviors. Llama-2 demonstrates a sharp threshold (τ = 0.25,0.32  beyond  α = -0.15), whereas Qwen2.5 displays continuous and higher levels of transfer (τ up to 0.61).