# Sustained Gradient Alignment Mediates Subliminal Learning in a Multi-Step Setting: Evidence from MNIST Auxiliary Logit Distillation Experiment

arXiv: https://arxiv.org/abs/2604.25779 · HTML: https://mlsanity.com/paper/2604.25779/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in On-policy and self-distillation for LLMs, one of 12 directions in Knowledge distillation.
- Builds on 5 of its 9 references that are in the map.
- First posted on arXiv 28 Apr 2026.

## Placed in

- Knowledge distillation / [On-policy and self-distillation for LLMs](https://mlsanity.com/distill/on-policy-and-self-distillation-for-llms/)

## Builds on (most cited first, 5 of 5)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [Subliminal Learning: Language models transmit behavioral traits via hidden signals in data](https://mlsanity.com/paper/2507.14805/) (2025)
- [A Comprehensive Survey on Knowledge Distillation](https://mlsanity.com/paper/2503.12067/) (2025)
- [How to Distill your BERT: An Empirical Study on the Impact of Weight Initialisation and Distillation Objectives](https://mlsanity.com/paper/2305.15032/) (2023)

## Built on by (most cited first, 7 of 7)

- [Learning Through Noise: Why Subliminal Learning Works and When It Fails](https://mlsanity.com/paper/2605.23645/) (2026)
- [Channel Location Constrains the Auditability of Subliminal Learning](https://mlsanity.com/paper/2606.22019/) (2026)
- [Scaling Model-Generated Distillation Data Can Make Latent Teacher Traits More Recoverable](https://mlsanity.com/paper/2608.26958/) (2026)
- [Functional compatibility as a determinant of persistent neural learning](https://mlsanity.com/paper/2608.22462/) (2026)
- [Towards Identifying the Dataset Biases Causing Phantom Transfer](https://mlsanity.com/paper/2609.14449/) (2026)
- [Why Ghost Outputs Teach: A Kernel-Based Understanding of Subliminal Learning](https://mlsanity.com/paper/2609.23260/) (2026)
- [Cross-Trait Transfer in Subliminal Learning](https://mlsanity.com/paper/2610.04260/) (2026)

## Abstract (arXiv)

In the MNIST auxiliary logit distillation experiment, a student can acquire an unintended teacher trait despite distilling only on no-class logits through a phenomenon called subliminal learning. Under a single-step gradient descent assumption, subliminal learning theory attributes this effect to alignment between the trait and distillation gradients, but does not guarantee that this alignment persists in a multi-step setting. We empirically show that gradient alignment remains weakly but consistently positive throughout training and causally contributes to trait acquisition. We show that a mitigation method called liminal training works by attenuating the alignment and fails to stop trait acquisition in this setup. These results suggest that mitigation methods that operate in this regime may not reliably suppress trait acquisition when the first-order drive dominates.