# Breaking the Tokenizer Barrier: On-Policy Distillation across Model Families

arXiv: https://arxiv.org/abs/2606.09456 · HTML: https://mlsanity.com/paper/2606.09456/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in On-policy and self-distillation for LLMs, one of 12 directions in Knowledge distillation.
- Builds on 20 of its 27 references that are in the map.
- First posted on arXiv 8 Jun 2026.

## Placed in

- Knowledge distillation / [On-policy and self-distillation for LLMs](https://mlsanity.com/distill/on-policy-and-self-distillation-for-llms/)

## Builds on (most cited first, 20 of 20)

- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models](https://mlsanity.com/paper/2403.13372/) (2024)
- [MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers](https://mlsanity.com/paper/2002.10957/) (2020)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes](https://mlsanity.com/paper/2306.13649/) (2023)
- [A Survey of On-Policy Distillation for Large Language Models](https://mlsanity.com/paper/2604.00626/) (2026)
- [Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?](https://mlsanity.com/paper/2603.24472/) (2026)
- [DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs](https://mlsanity.com/paper/2503.07067/) (2025)
- [Efficient Transformers with Dynamic Token Pooling](https://mlsanity.com/paper/2211.09761/) (2022)
- [Universal Cross-Tokenizer Distillation via Approximate Likelihood Matching](https://mlsanity.com/paper/2503.20083/) (2025)
- [A Formal Perspective on Byte-Pair Encoding](https://mlsanity.com/paper/2306.16837/) (2023)
- [Multi-Level Optimal Transport for Universal Cross-Tokenizer Knowledge Distillation on Language Models](https://mlsanity.com/paper/2412.14528/) (2024)
- [PACED: Distillation and On-Policy Self-Distillation at the Frontier of Student Competence](https://mlsanity.com/paper/2603.11178/) (2026)
- [ToDi: Token-wise Distillation via Fine-Grained Divergence Control](https://mlsanity.com/paper/2505.16297/) (2025)
- [AlignDistil: Token-Level Language Model Alignment as Adaptive Policy Distillation](https://mlsanity.com/paper/2503.02832/) (2025)
- [Contextual Distillation Model for Diversified Recommendation](https://mlsanity.com/paper/2406.09021/) (2024)

## Built on by (most cited first, 9 of 9)

- [Cliff: Learning Process Rewards from the First Mistake](https://mlsanity.com/paper/2609.02817/) (2026)
- [From Parameters to Behaviors: A Survey of Model Fusion for Large Language Models](https://mlsanity.com/paper/2609.19553/) (2026)
- [Mint-Agent: Introducing Finance-Native Agentic Foundation Models](https://mlsanity.com/paper/2608.16386/) (2026)
- [CompassOPD: Cross-Family On-Policy Distillation via Within-Family Likelihood Shifts](https://mlsanity.com/paper/2609.10154/) (2026)
- [Allspark: Weak to Strong Transfer via Alternating Chain of Thought](https://mlsanity.com/paper/2609.32913/) (2026)
- [Latent-MOPD: Latent Multi-Teacher On-Policy Distillation](https://mlsanity.com/paper/2610.02381/) (2026)
- [CAST: Credit Alignment for Cross-Tokenizer On-Policy Distillation](https://mlsanity.com/paper/2610.13257/) (2026)
- [Imprinting: Knowledge Distillation Can Shift Student Behavior](https://mlsanity.com/paper/2610.18517/) (2026)
- [ActionOPD: Distilling Decisions, Not Tokens](https://mlsanity.com/paper/2610.19820/) (2026)

## Abstract (arXiv)

On-Policy Distillation (OPD) has become a core technique in the post-training of Large Language Models (LLMs) for transferring knowledge from domain experts to student models. However, existing OPD distillation methods require teacher and student models to share the same tokenizer, restricting the applicability of OPD within the model series. Current mainstream practice typically employs Supervised Fine-Tuning (SFT) on teacher-generated responses for cross-tokenizer distillation, which fails to capture the rich knowledge embedded in the teacher's probability distribution. In this work, we enable the standard on-policy distillation method to operate across model families, ensuring that high-fidelity token-level signals can propagate across different tokenizers with a precise token-mapping algorithm. Extensive experiments show that cross-tokenizer OPD is significantly more compute-efficient than baselines on various benchmarks. Our results unlock a broader range of teacher-student pairs for OPD, opening up new avenues for adapting and enhancing interactions between LLMs.