# On Teacher Hacking in Language Model Distillation

arXiv: https://arxiv.org/abs/2502.02671 · HTML: https://mlsanity.com/paper/2502.02671/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Reward hacking and misspecification, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- First posted on arXiv 4 Feb 2025.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Reward hacking and misspecification](https://mlsanity.com/alignment/reward-hacking-and-misspecification/)
- Knowledge distillation / [Knowledge distillation for LLMs](https://mlsanity.com/distill/knowledge-distillation-for-llms/)

## Built on by (most cited first, 6 of 6)

- [Universal Cross-Tokenizer Distillation via Approximate Likelihood Matching](https://mlsanity.com/paper/2503.20083/) (2025)
- [Supervised Reinforcement Learning: From Expert Trajectories to Step-wise Reasoning](https://mlsanity.com/paper/2510.25992/) (2025)
- [MASSV: Multimodal Adaptation and Self-Data Distillation for Speculative Decoding of Vision-Language Models](https://mlsanity.com/paper/2505.10526/) (2025)
- [Enhancing Reasoning Capabilities in SLMs with Reward Guided Dataset Distillation](https://mlsanity.com/paper/2507.00054/) (2025)
- [Scaling Properties of Same-Family On-Policy Distillation](https://mlsanity.com/paper/2609.32722/) (2026)
- [How Should Teachers Be Prepared? RL on Student-Induced States for On-Policy Distillation](https://mlsanity.com/paper/2610.04950/) (2026)

## Abstract (arXiv)

Post-training of language models (LMs) increasingly relies on the following two stages: (i) knowledge distillation, where the LM is trained to imitate a larger teacher LM, and (ii) reinforcement learning from human feedback (RLHF), where the LM is aligned by optimizing a reward model. In the second RLHF stage, a well-known challenge is reward hacking, where the LM over-optimizes the reward model. Such phenomenon is in line with Goodhart's law and can lead to degraded performance on the true objective. In this paper, we investigate whether a similar phenomenon, that we call teacher hacking, can occur during knowledge distillation. This could arise because the teacher LM is itself an imperfect approximation of the true distribution. To study this, we propose a controlled experimental setup involving: (i) an oracle LM representing the ground-truth distribution, (ii) a teacher LM distilled from the oracle, and (iii) a student LM distilled from the teacher. Our experiments reveal the following insights. When using a fixed offline dataset for distillation, teacher hacking occurs; moreover, we can detect it by observing when the optimization process deviates from polynomial convergence laws. In contrast, employing online data generation techniques effectively mitigates teacher hacking. More precisely, we identify data diversity as the key factor in preventing hacking. Overall, our findings provide a deeper understanding of the benefits and limitations of distillation for building robust and efficient LMs.