# Optimizing Knowledge Distillation in Transformers: Enabling Multi-Head Attention without Alignment Barriers

arXiv: https://arxiv.org/abs/2502.07436 · HTML: https://mlsanity.com/paper/2502.07436/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- First posted on arXiv 11 Feb 2025.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Built on by (most cited first, 3 of 3)

- [Integrating Inductive Biases in Transformers via Distillation for Financial Time Series Forecasting](https://mlsanity.com/paper/2603.16985/) (2026)
- [RestoRect: Degraded Image Restoration via Latent Rectified Flow & Feature Distillation](https://mlsanity.com/paper/2509.23480/) (2025)
- [MANAR: Memory-augmented Attention with Navigational Abstract Conceptual Representation](https://mlsanity.com/paper/2603.18676/) (2026)

## Abstract (arXiv)

Knowledge distillation (KD) in transformers often faces challenges due to misalignment in the number of attention heads between teacher and student models. Existing methods either require identical head counts or introduce projectors to bridge dimensional gaps, limiting flexibility and efficiency. We propose Squeezing-Heads Distillation (SHD), a novel approach that enables seamless knowledge transfer between models with varying head counts by compressing multi-head attention maps via efficient linear approximation. Unlike prior work, SHD eliminates alignment barriers without additional parameters or architectural modifications. Our method dynamically approximates the combined effect of multiple teacher heads into fewer student heads, preserving fine-grained attention patterns while reducing redundancy. Experiments across language (LLaMA, GPT) and vision (DiT, MDT) generative and vision (DeiT) discriminative tasks demonstrate SHD's effectiveness: it outperforms logit-based and feature-alignment KD baselines, achieving state-of-the-art results in image classification, image generation language fine-tuning, and language pre-training. The key innovations of flexible head compression, projector-free design, and linear-time complexity make SHD a versatile and scalable solution for distilling modern transformers. This work bridges a critical gap in KD, enabling efficient deployment of compact models without compromising performance.