mlsanityData to 7 Oct 2026

Turning the TIDE: Cross-Architecture Distillation for Diffusion Large Language Models

arXiv 2604.26951 ↗First posted 29 Apr 2026cs.CL cs.AI cs.LGcited by 0

On the map: Efficient LLM inference and serving → Diffusion LLM and MoE inference

Abstract (arXiv)

Diffusion large language models (dLLMs) offer parallel decoding and bidirectional context, but state-of-the-art dLLMs require billions of parameters for competitive performance. While existing distillation methods for dLLMs reduce inference steps within a single architecture, none address cross-architecture knowledge transfer, in which the teacher and student differ in architecture, attention mechanism, and tokenizer. We present TIDE, the first framework for cross-architecture dLLM distillation, comprising three modular components: (1) TIDAL, which jointly modulates distillation strength across training progress and diffusion timestep to account for the teacher's noise-dependent reliability; (2) CompDemo, which enriches the teacher's context via complementary mask splitting to improve predictions under heavy masking; and (3) Reverse CALM, a cross-tokenizer objective that inverts chunk-level likelihood matching, yielding bounded gradients and dual-end noise filtering. Distilling 8B dense and 16B MoE teachers into a 0.6B student via two heterogeneous pipelines outperforms the baseline by an average of 1.53 points across eight benchmarks, yielding notable gains in code generation, where HumanEval scores reach 48.78 compared to 32.3 for the AR baseline.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Diffusion LLM and MoE inference, one of 12 directions in Efficient LLM inference and serving. Builds on 34 of its 36 references that are in the map. First posted on arXiv 29 Apr 2026.

Builds on

34 of 36 references.

References · 34

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2025Qwen3 Technical Reportcited by 8,195
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2024Qwen2.5 Technical Reportcited by 4,713
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  10. 2021Program Synthesis with Large Language Modelscited by 3,509
  11. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  12. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
22 more
  1. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  2. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  3. 2025Large Language Diffusion Modelscited by 872
  4. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  5. 2024Simple and Effective Masked Diffusion Language Modelscited by 800
  6. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distributioncited by 712
  7. 2025Dream 7B: Diffusion Large Language Modelscited by 515
  8. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Modelscited by 385
  9. 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 307
  10. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Modelscited by 268
  11. 2026Self-Distillation Enables Continual Learningcited by 212
  12. 2025LLaDA2.0: Scaling Up Diffusion Language Models to 100Bcited by 154
  13. 2024Beyond Autoregression: Fast LLMs via Self-Distillation Through Timecited by 73
  14. 2024Distillation of Discrete Diffusion through Dimensional Correlationscited by 46
  15. 2025Universal Cross-Tokenizer Distillation via Approximate Likelihood Matchingcited by 40
  16. 2025WeDLM: Reconciling Diffusion Language Models with Standard Causal Attention for Fast Inferencecited by 38
  17. 2024Zero-Shot Tokenizer Transfercited by 30
  18. 2024Dual-Space Knowledge Distillation for Large Language Modelscited by 25
  19. 2025CDLM: Consistency Diffusion Language Models For Faster Samplingcited by 20
  20. 2025TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Modelscited by 19
  21. 2025Learnable Sampler Distillation for Discrete Diffusion Modelscited by 8
  22. 2025What Should Feature Distillation Transfer in LLMs? A Task-Tangent Geometry Viewcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Finish First, Perfect Later: Test-Time Token-Level Cross-Validation for Diffusion Large Language Models
  2. 2026Accelerating Masked Diffusion Large Language Models: A Survey of Efficient Inference Techniques
  3. 2026Diffusion Language Models: An Experimental Analysis
  4. 2026Temporal Self-Distillation: Faster Inference in Discrete Diffusion Language Models
  5. 2026CAI-DLLM: Convergence Aware Inference for Diffusion Language Models
  6. 2026ES-dLLM: Efficient Inference for Diffusion Large Language Models by Early-Skipping
  7. 2026Divide and Conquer: Accelerating Diffusion-Based Large Language Models via Adaptive Parallel Decoding
  8. 2026dLLM: Simple Diffusion Language Modeling