mlsanityData to 7 Oct 2026

Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs

arXiv 2402.12030 ↗First posted 19 Feb 2024cs.CLcited by 61

Not on the map. Why?

Abstract (arXiv)

Deploying large language models (LLMs) of several billion parameters can be impractical in most industrial use cases due to constraints such as cost, latency limitations, and hardware accessibility. Knowledge distillation (KD) offers a solution by compressing knowledge from resource-intensive large models to smaller ones. Various strategies exist, some relying on the text generated by the teacher model and optionally utilizing his logits to enhance learning. However, these methods based on logits often require both teacher and student models to share the same tokenizer, limiting their applicability across different LLM families. In this paper, we introduce Universal Logit Distillation (ULD) loss, grounded in optimal transport, to address this limitation. Our experimental results demonstrate the effectiveness of ULD loss in enabling distillation across models with different architectures and tokenizers, paving the way to a more widespread use of distillation techniques.

Key facts. Cited by 61 papers in the map (data through 7 Oct 2026). Builds on 42 of its 65 references that are in the map. First posted on arXiv 19 Feb 2024.

Builds on

42 of 65 references.

References · 42

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  7. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  9. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  10. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,277
  11. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  12. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
30 more
  1. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  2. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  3. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  4. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  5. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  6. 2016Sequence-Level Knowledge Distillationcited by 761
  7. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  8. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  9. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
  10. 2023The Falcon Series of Open Language Modelscited by 526
  11. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  12. 2019Generalizing from a Few Examples: A Survey on Few-Shot Learningcited by 462
  13. 2023A Survey of Hallucination in Large Foundation Modelscited by 365
  14. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 240
  15. 2019A Study of BFLOAT16 for Deep Learning Trainingcited by 227
  16. 2023LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructionscited by 124
  17. 2022Fantastic Questions and Where to Find Them: FairytaleQA -- An Authentic Dataset for Narrative Comprehensioncited by 79
  18. 2023Baby Llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penaltycited by 57
  19. 2020QED: A Framework and Dataset for Explanations in Question Answeringcited by 55
  20. 2021Automatic Text Evaluation through the Lens of Wasserstein Barycenterscited by 38
  21. 2015Fast Discrete Distribution Clustering Using Wasserstein Barycenter with Sparse Supportcited by 30
  22. 2018Distilled Wasserstein Learning for Word Embedding and Topic Modelingcited by 18
  23. 2022Wasserstein K-means for clustering probability distributionscited by 16
  24. 2022SMaLL-100: Introducing Shallow Multilingual Machine Translation Model for Low-Resource Languagescited by 16
  25. 2023An Empirical Comparison of LM-based Question and Answer Generation Methodscited by 15
  26. 2023TeacherLM: Teaching to Fish Rather Than Giving the Fish, Language Modeling Likewisecited by 15
  27. 2022Optimal Transport for Unsupervised Hallucination Detection in Neural Machine Translationcited by 13
  28. 2022Reducing Retraining by Recycling Parameter-Efficient Promptscited by 11
  29. 2023Revisiting Instruction Fine-tuned Model Evaluation to Guide Industrial Applicationscited by 11
  30. 2021KNOT: Knowledge Distillation using Optimal Transport for Solving NLP Taskscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 21; 2026: 35.

61 papers.

Sort:

By area:

  1. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  2. 2026Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixescited by 122
  3. 2026Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervisioncited by 58
  4. 2025Universal Cross-Tokenizer Distillation via Approximate Likelihood Matchingcited by 40
  5. 2024Multi-Level Optimal Transport for Universal Cross-Tokenizer Knowledge Distillation on Language Modelscited by 35
  6. 2024A Survey of Small Language Modelscited by 31
  7. 2024MiniPLM: Knowledge Distillation for Pre-Training Language Modelscited by 27
  8. 2026PACED: Distillation and On-Policy Self-Distillation at the Frontier of Student Competencecited by 26
  9. 2024Dual-Space Knowledge Distillation for Large Language Modelscited by 25
  10. 2025Sparse Logit Sampling: Accelerating Knowledge Distillation in LLMscited by 14
  11. 2025InfiGFusion: Graph-on-Logits Distillation via Efficient Gromov-Wasserstein for Model Fusioncited by 11
  12. 2026SimCT: Recovering Lost Supervision for Cross-Tokenizer On-Policy Distillationcited by 11
38 more
  1. 2025Smoothing Out Hallucinations: Mitigating LLM Hallucination with Smoothed Knowledge Distillationcited by 10
  2. 2025GenRecal: Generation after Recalibration from Large to Small Vision-Language Modelscited by 9
  3. 2025A Survey on Efficient Large Language Model Training: From Data-centric Perspectivescited by 9
  4. 2026Multi-Rollout On-Policy Distillation via Peer Successes and Failurescited by 8
  5. 2025Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Modelscited by 6
  6. 2025Linear Representation Transferability Hypothesis: Leveraging Small Models to Steer Large Modelscited by 6
  7. 2025Masking Teacher and Reinforcing Student for Distilling Vision-Language Modelscited by 6
  8. 2026Cross-Tokenizer LLM Distillation through a Byte-Level Interfacecited by 6
  9. 2025InfiFusion: A Unified Framework for Enhanced Cross-Model Reasoning via LLM Fusioncited by 5
  10. 2025Every Expert Matters: Towards Effective Knowledge Distillation for Mixture-of-Experts Language Modelscited by 5
  11. 2025A Dual-Space Framework for General Knowledge Distillation of Large Language Modelscited by 5
  12. 2025Cross-Tokenizer Likelihood Scoring Algorithms for Language Model Distillationcited by 5
  13. 2026DWA-KD: Dual-Space Weighting and Time-Warped Alignment for Cross-Tokenizer Knowledge Distillationcited by 5
  14. 2026KDFlow: A User-Friendly and Efficient Knowledge Distillation Framework for Large Language Modelscited by 5
  15. 2026A Formula-Driven Survey and Research Agenda for On-Policy Distillationcited by 5
  16. 2026Cross-Tokenizer On-Policy Distillation via Byte-Prefix Marginalizationcited by 5
  17. 2025CoT2Align: Cross-Chain of Thought Distillation via Optimal Transport Alignment for Language Models with Different Tokenizerscited by 4
  18. 2025Overcoming Vocabulary Mismatch: Vocabulary-agnostic Teacher Guided Language Modelingcited by 4
  19. 2025ExpertSteer: Intervening in LLMs through Expert Knowledgecited by 4
  20. 2025Training-Free Tokenizer Transplantation via Orthogonal Matching Pursuitcited by 4
  21. 2025OmniDraft: A Cross-vocabulary, Online Adaptive Drafter for On-device Speculative Decodingcited by 4
  22. 2026BidirLM: From Text to Omnimodal Bidirectional Encoders by Adapting and Composing Causal LLMscited by 4
  23. 2025Probabilistic Token Alignment for Large Language Model Fusioncited by 3
  24. 2026SRA: Span Representation Alignment for Large Language Model Distillationcited by 3
  25. 2026Self-Policy Distillation via Capability-Selective Subspace Projectioncited by 3
  26. 2026EasyOPD: An Easy-to-use On-Policy Distillation Framework for Large Language Modelscited by 3
  27. 2025Punching Above Precision: Small Quantized Model Distillation with Learnable Regularizercited by 2
  28. 2026KD4MT: A Survey of Knowledge Distillation for Machine Translationcited by 2
  29. 2026X-Token: Projection-Guided Cross-Tokenizer Knowledge Distillationcited by 2
  30. 2026Dual-Space Knowledge Distillation with Key-Query Matching for Large Language Models with Vocabulary Mismatchcited by 1
  31. 2026Polyglot Teachers: Evaluating Language Models for Multilingual Synthetic Data Generationcited by 1
  32. 2026Cliff: Learning Process Rewards from the First Mistakecited by 1
  33. 2025Multi-Hypothesis Distillation of Multilingual Neural Translation Models for Low-Resource Languagescited by 0
  34. 2026Advancing Model Refinement: Muon-Optimized Distillation and Quantization for LLM Deploymentcited by 0
  35. 2026Mixture-of-Models: Unifying Heterogeneous Agents via N-Way Self-Evaluating Deliberationcited by 0
  36. 2026Detecting Non-Membership in LLM Training Data via Rank Correlationscited by 0
  37. 2026Can MLLMs Reason About Visual Persuasion? Evaluating the Efficacy and Faithfulness of Reasoningcited by 0
  38. 2026EVL-ECG: Efficient ECG Interpretation With Multi-Aspect Heterogeneous Knowledge Distillationcited by 0

50 most cited of 61 All 61 (JSON)