# Why Ghost Outputs Teach: A Kernel-Based Understanding of Subliminal Learning

arXiv: https://arxiv.org/abs/2609.23260 · HTML: https://mlsanity.com/paper/2609.23260/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 29 references that are in the map.
- First posted on arXiv 20 Sep 2026.

## Builds on (most cited first, 20 of 20)

- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution](https://mlsanity.com/paper/2202.10054/) (2022)
- [Estimating Training Data Influence by Tracing Gradient Descent](https://mlsanity.com/paper/2002.08484/) (2020)
- [On Exact Computation with an Infinitely Wide Neural Net](https://mlsanity.com/paper/1904.11955/) (2019)
- [LESS: Selecting Influential Data for Targeted Instruction Tuning](https://mlsanity.com/paper/2402.04333/) (2024)
- [Finite Depth and Width Corrections to the Neural Tangent Kernel](https://mlsanity.com/paper/1909.05989/) (2019)
- [Learning Dynamics of LLM Finetuning](https://mlsanity.com/paper/2407.10490/) (2024)
- [Subliminal Learning: Language models transmit behavioral traits via hidden signals in data](https://mlsanity.com/paper/2507.14805/) (2025)
- [Stiffness: A New Perspective on Generalization in Neural Networks](https://mlsanity.com/paper/1901.09491/) (2019)
- [Introduction to Core-sets: an Updated Survey](https://mlsanity.com/paper/2011.09384/) (2020)
- [The Local Elasticity of Neural Networks](https://mlsanity.com/paper/1910.06943/) (2019)
- [Towards Understanding Subliminal Learning: When and How Hidden Biases Transfer](https://mlsanity.com/paper/2509.23886/) (2025)
- [Better Supervisory Signals by Observing Learning Paths](https://mlsanity.com/paper/2203.02485/) (2022)
- [Subliminal Effects in Your Data: A General Mechanism via Log-Linearity](https://mlsanity.com/paper/2602.04863/) (2026)
- [Subliminal Steering: Stronger Encoding of Hidden Signals](https://mlsanity.com/paper/2604.25783/) (2026)
- [From Data to Behavior: Predicting Unintended Model Behaviors Before Training](https://mlsanity.com/paper/2602.04735/) (2026)
- [Sustained Gradient Alignment Mediates Subliminal Learning in a Multi-Step Setting: Evidence from MNIST Auxiliary Logit Distillation Experiment](https://mlsanity.com/paper/2604.25779/) (2026)
- [lpNTK: Better Generalisation with Less Data via Sample Interaction During Learning](https://mlsanity.com/paper/2401.08808/) (2024)
- [Subliminal Signals in Preference Labels](https://mlsanity.com/paper/2603.01204/) (2026)
- [Learning Dynamics of Zeroth-Order Optimization: A Kernel Perspective](https://mlsanity.com/paper/2605.03373/) (2026)

## Abstract (arXiv)

Subliminal Learning (SL) is a recently identified phenomenon in which a student model acquires downstream task capabilities by matching seemingly unrelated auxiliary outputs from a teacher, despite never observing task labels, task-specific outputs, or the original training data. While recent studies have identified where subliminal signals may reside, the optimization mechanism underlying this phenomenon remains poorly understood. In this work, we provide a mechanistic understanding of SL through the lens of learning dynamics. Specifically, we derive a chained cross-task kernel that explicitly links ghost-output supervision to changes in task predictions through shared backbone representations. Our unified analytical framework provides a rigorous mathematical explanation for three central empirical puzzles in SL: (i) under shared initialization, the transfer operator forms a strictly Positive Semi-Definite (PSD) structure, guaranteeing that ghost-output optimization aligns the student with the teacher's true task objective without explicit label exposure; (ii) the ghost-output dimensionality acts as an explicit rank bottleneck governing the transfer of task-relevant features; and (iii) synthetic, high-entropy inputs function as broadband probes that maximize cross-task kernel overlap, explaining why random noise consistently outperforms structured data for subliminal transfer. Experiments on the canonical ghost-output setting validate all three theoretical predictions, providing the first learning-dynamics-based theoretical explanation of how ghost-output supervision gives rise to subliminal learning.