# ToDi: Token-wise Distillation via Fine-Grained Divergence Control

arXiv: https://arxiv.org/abs/2505.16297 · HTML: https://mlsanity.com/paper/2505.16297/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation.
- First posted on arXiv 22 May 2025.

## Placed in

- Knowledge distillation / [Knowledge distillation for LLMs](https://mlsanity.com/distill/knowledge-distillation-for-llms/)

## Built on by (most cited first, 27 of 27)

- [Entropy-Aware On-Policy Distillation of Language Models](https://mlsanity.com/paper/2603.07079/) (2026)
- [SCOPE: Signal-Calibrated On-Policy Distillation Enhancement with Dual-Path Adaptive Weighting](https://mlsanity.com/paper/2604.10688/) (2026)
- [When Are Teacher Tokens Reliable? Position-Weighted On-Policy Self-Distillation for Reasoning](https://mlsanity.com/paper/2605.21606/) (2026)
- [SelecTKD: Selective Token-Weighted Knowledge Distillation for LLMs](https://mlsanity.com/paper/2510.24021/) (2025)
- [OVD: On-policy Verbal Distillation](https://mlsanity.com/paper/2601.21968/) (2026)
- [Learn where to Click from Yourself: On-Policy Self-Distillation for GUI Grounding](https://mlsanity.com/paper/2605.00642/) (2026)
- [Multi-Rollout On-Policy Distillation via Peer Successes and Failures](https://mlsanity.com/paper/2605.12652/) (2026)
- [Breaking the Tokenizer Barrier: On-Policy Distillation across Model Families](https://mlsanity.com/paper/2606.09456/) (2026)
- [Backtracking When It Strays: Mitigating Dual Exposure Biases in LLM Reasoning Distillation](https://mlsanity.com/paper/2605.19433/) (2026)
- [A Formula-Driven Survey and Research Agenda for On-Policy Distillation](https://mlsanity.com/paper/2606.22793/) (2026)
- [RISE: Recursive Improvement via Self-Extrapolating Policy Distillation](https://mlsanity.com/paper/2609.05295/) (2026)
- [When Top-K Misses the Decision: Tool-Call Drift in Multi-Teacher On-Policy Distillation](https://mlsanity.com/paper/2607.07050/) (2026)
- [EGAD: Entropy-Guided Adaptive Distillation for Token-Level Knowledge Transfer](https://mlsanity.com/paper/2605.01732/) (2026)
- [SCOPE-OPSD: Fisher-Conditioned Privileged Subspaces for On-Policy Self-Distillation](https://mlsanity.com/paper/2609.12579/) (2026)
- [How Does Prefix Matter in Reasoning Model Tuning?](https://mlsanity.com/paper/2601.01624/) (2026)
- [Distribution Corrected Offline Data Distillation for Large Language Models](https://mlsanity.com/paper/2605.14071/) (2026)
- [The Bridge-Garden Dilemma in LLM Distillation: Why Mixing Hard and Soft Labels Works](https://mlsanity.com/paper/2605.26246/) (2026)
- [Rethinking Reverse KL as Adaptive Entropy Distillation](https://mlsanity.com/paper/2608.14685/) (2026)
- [FOCUS & RePAIR: Mitigating Text Degeneration via Token-Level Guidance for Pruned Large Language Models](https://mlsanity.com/paper/2608.26676/) (2026)
- [ACTD: Anchor-Based Cross-Tokenizer Distillation with Residual Regularization](https://mlsanity.com/paper/2608.29662/) (2026)
- [A Unified Per-Token Gating Family for On-Policy Distillation: FKL/RKL Mixing with Multi-Channel and Bias Coefficients](https://mlsanity.com/paper/2609.11768/) (2026)
- [OSPD: On-Policy Self-Distillation for Persona-Consistent Dialogue](https://mlsanity.com/paper/2609.34418/) (2026)
- [ReTaCo: Residual-Target Control for On-Policy Distillation](https://mlsanity.com/paper/2609.39275/) (2026)
- [How Should Teachers Be Prepared? RL on Student-Induced States for On-Policy Distillation](https://mlsanity.com/paper/2610.04950/) (2026)
- [Outcome-Guided On-Policy Self-Distillation](https://mlsanity.com/paper/2610.05070/) (2026)
- [Transfer-Stratified On-Policy Distillation for RL-Improved Reasoning Teachers](https://mlsanity.com/paper/2610.05974/) (2026)
- [CAST: Credit Alignment for Cross-Tokenizer On-Policy Distillation](https://mlsanity.com/paper/2610.13257/) (2026)

## Abstract (arXiv)

Large language models (LLMs) offer impressive performance but are impractical for resource-constrained deployment due to high latency and energy consumption. Knowledge distillation (KD) addresses this by transferring knowledge from a large teacher to a smaller student model. However, conventional KD, notably approaches like Forward KL (FKL) and Reverse KL (RKL), apply uniform divergence loss across the entire vocabulary, neglecting token-level prediction discrepancies. By investigating these representative divergences via gradient analysis, we reveal that FKL boosts underestimated tokens, while RKL suppresses overestimated ones, showing their complementary roles. Based on this observation, we propose Token-wise Distillation (ToDi), a novel method that adaptively combines FKL and RKL per token using a sigmoid-based weighting function derived from the teacher-student probability log-ratio. ToDi dynamically emphasizes the appropriate divergence for each token, enabling precise distribution alignment. We demonstrate that ToDi consistently outperforms recent distillation baselines using uniform or less granular strategies across instruction-following benchmarks. Extensive ablation studies and efficiency analysis further validate ToDi's effectiveness and practicality.