mlsanityData to 7 Oct 2026

Dual-Vocabulary Language Model for Cross-Tokenizer Distillation

arXiv 2609.33816 ↗First posted 27 Sep 2026cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

On-policy distillation (OPD) bridges teacher supervision and student behavior, but different teacher-student tokenizers introduce misalignment in both input tokenization (#1) and output logits (#2). Existing approaches address the former by matching same-text spans or converting tokens to bytes, often losing fine-grained token information or disrupting the native-token paradigm, while for the latter, strategies such as ranking, padding, or key-token selection retain only shared logit dimensions, resulting in much distribution loss. In this paper, we propose Dual-Vocabulary Language Model (DVLM), which replaces the teacher's LM head with a new student-vocabulary projection head and obtains full-dimensional student logits (for #2). To support student tokens (for #1), it takes a Parallel-Tokenized Sequence (PTS) as input, which concatenates the original teacher-tokenized sequence and a re-tokenized sequence formed by independently converting each student token into a teacher-token group. To avoid inference inconsistency with the original teacher tokens, the Hybrid-Prefix Attention (HPA) further restricts re-tokenized groups to their corresponding teacher prefix and uses its last state as the aggregation of the original student-token representation for projection into the student vocabulary space. Similarly, via the combined use of PTS and HPA, the DVLM teacher can provide distribution-aligned supervision with the student's input-tokenization and output-logit during OPD. Experimental results demonstrate that our DVLM teacher has a similar converged loss as the original teacher model and enables student models to improve performance across six reasoning tasks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 41 of its 53 references that are in the map. First posted on arXiv 27 Sep 2026.

Builds on

41 of 53 references.

References · 41

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2023A Survey of Large Language Modelscited by 2,537
  9. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  10. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  11. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  12. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
29 more
  1. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  2. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  3. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  4. 2026A Survey of On-Policy Distillation for Large Language Modelscited by 148
  5. 2023D4: Improving LLM Pretraining via Document De-Duplication and Diversificationcited by 146
  6. 2024DistiLLM: Towards Streamlined Distillation for Large Language Modelscited by 145
  7. 2023Tokenizer Choice For LLM Training: Negligible or Crucial?cited by 105
  8. 2025LeetCodeDataset: A Temporal Dataset for Robust Evaluation and Efficient Training of Code LLMscited by 67
  9. 2024Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMscited by 61
  10. 2025Universal Cross-Tokenizer Distillation via Approximate Likelihood Matchingcited by 40
  11. 2024Multi-Level Optimal Transport for Universal Cross-Tokenizer Knowledge Distillation on Language Modelscited by 35
  12. 2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Modelscited by 31
  13. 2024Survey of different Large Language Model Architectures: Trends, Benchmarks, and Challengescited by 31
  14. 2024Dual-Space Knowledge Distillation for Large Language Modelscited by 25
  15. 2024Gold Panning in Vocabulary: An Adaptive Method for Vocabulary Expansion of Domain-Specific LLMscited by 11
  16. 2025TokAlign: Efficient Vocabulary Adaptation via Token Alignmentcited by 11
  17. 2026SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillationcited by 11
  18. 2025Enhancing Cross-Tokenizer Knowledge Distillation with Contextual Dynamical Mappingcited by 10
  19. 2025A Survey of Inductive Reasoning for Large Language Modelscited by 8
  20. 2026Cross-Tokenizer LLM Distillation through a Byte-Level Interfacecited by 6
  21. 2025Cross-Tokenizer Likelihood Scoring Algorithms for Language Model Distillationcited by 5
  22. 2026A Formula-Driven Survey and Research Agenda for On-Policy Distillationcited by 5
  23. 2026ReNIO: Reweighting Negative Trajectory Importance for LLM On-Policy Distillationcited by 5
  24. 2026Cross-Tokenizer On-Policy Distillation via Byte-Prefix Marginalizationcited by 5
  25. 2025CoT2Align: Cross-Chain of Thought Distillation via Optimal Transport Alignment for Language Models with Different Tokenizerscited by 4
  26. 2025Overcoming Vocabulary Mismatch: Vocabulary-agnostic Teacher Guided Language Modelingcited by 4
  27. 2025Training-Free Tokenizer Transplantation via Orthogonal Matching Pursuitcited by 4
  28. 2026SRA: Span Representation Alignment for Large Language Model Distillationcited by 3
  29. 2026X-Token: Projection-Guided Cross-Tokenizer Knowledge Distillationcited by 2

Built on by

Not cited yet.