# Enhancing Cross-Tokenizer Knowledge Distillation with Contextual Dynamical Mapping

arXiv: https://arxiv.org/abs/2502.11104 · HTML: https://mlsanity.com/paper/2502.11104/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge distillation for LLMs, one of 12 directions in Knowledge distillation.
- First posted on arXiv 16 Feb 2025.

## Placed in

- Knowledge distillation / [Knowledge distillation for LLMs](https://mlsanity.com/distill/knowledge-distillation-for-llms/)

## Built on by (most cited first, 10 of 10)

- [SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillation](https://mlsanity.com/paper/2605.07711/) (2026)
- [ProxyThinker: Test-Time Guidance through Small Visual Reasoners](https://mlsanity.com/paper/2505.24872/) (2025)
- [KDFlow: A User-Friendly and Efficient Knowledge Distillation Framework for Large Language Models](https://mlsanity.com/paper/2603.01875/) (2026)
- [SimpleOPD: Simple Tokenizer-Agnostic On-Policy Distillation for Long-Context Reasoning](https://mlsanity.com/paper/2608.14277/) (2026)
- [Cross-Tokenizer Likelihood Scoring Algorithms for Language Model Distillation](https://mlsanity.com/paper/2512.14954/) (2025)
- [DWA-KD: Dual-Space Weighting and Time-Warped Alignment for Cross-Tokenizer Knowledge Distillation](https://mlsanity.com/paper/2602.21669/) (2026)
- [Dual-Space Knowledge Distillation with Key-Query Matching for Large Language Models with Vocabulary Mismatch](https://mlsanity.com/paper/2603.22056/) (2026)
- [Detecting Non-Membership in LLM Training Data via Rank Correlations](https://mlsanity.com/paper/2603.22707/) (2026)
- [ACTD: Anchor-Based Cross-Tokenizer Distillation with Residual Regularization](https://mlsanity.com/paper/2608.29662/) (2026)
- [Dual-Vocabulary Language Model for Cross-Tokenizer Distillation](https://mlsanity.com/paper/2609.33816/) (2026)

## Abstract (arXiv)

Knowledge Distillation (KD) has emerged as a prominent technique for model compression. However, conventional KD approaches primarily focus on homogeneous architectures with identical tokenizers, constraining their applicability in cross-architecture scenarios. As for the cross-tokenizer KD, the differences in the tokenizers give rise to two fundamental challenges: (1) sequence misalignment caused by divergent tokenization strategies, and (2) mismatched vocabulary size and composition. While existing probability-matching methods attempt to address these issues, their efficacy remains limited due to suboptimal alignment in both the sequence and vocabulary aspects. To overcome these limitations, we propose Contextual Dynamic Mapping (CDM), a novel cross-tokenizer distillation framework that employs contextual information to enhance sequence alignment precision and dynamically improves vocabulary mapping. We evaluated the effectiveness of our approach across five advanced and widely-used model families (i.e, LLama3, Phi3, Gemma2, OPT and Qwen2), which were configured into three distinct teacher-student pairs. Our method shows significant advantages over existing cross-tokenizer distillation baselines across diverse benchmarks, including instruction-following, code generation and math. Notably, our analysis reveals that combining conventional same-tokenizer distillation and cross-tokenizer distillation through CDM yields further performance improvements. The code is available at https://github.com/pppa2019/ContexualDynamicMapping