mlsanityData to 14 Oct 2026

Context-Tower Conversion Preserves Generation While Freezing Retains Knowledge: Low-Budget AR-to-Diffusion Conversion of MoE LLMs

arXiv 2610.02657 ↗First posted 2 Oct 2026cs.LGcited by 0

On the map: Diffusion models → Discrete and text diffusion

also in 1 more
Abstract (arXiv)

Converting a pretrained autoregressive (AR) model to a diffusion language model (dLLM) enables parallel generation without pretraining a new model. Published conversion methods differ by roughly three orders of magnitude in training data and have not been compared under a common protocol. We compare two conversions of the same 30B Mixture-of-Experts (MoE) parent, holding the corpus, supervised-token budget, trainable parameter set and evaluation harness fixed, each under its own training recipe. The in-place model updates a subset of the parent's weights using denoising and representation-alignment losses; the frozen-tower model instead conditions through cross-attention on a frozen causal copy of the parent. With 1B training tokens, the frozen-tower model scores 71.60 on HumanEval pass@10 against 6.19 for the in-place model, an 11.6x improvement. At the same budget it also keeps 95% of the parent's GSM8K score and 99% of its MMLU-Pro score. A dense-parent experiment reproduces the HumanEval separation. Within the two-tower design at about 500M tokens, freezing the context tower retains substantially more MMLU-Pro performance than training it, while both give similar observed HumanEval scores. Our theoretical analysis establishes that both conversion classes contain an exact sampler for the AR parent under a hard attention mask and left-to-right commitment of one position per round. Under a shared loss, freezing removes the gradient contribution through the context states. Furthermore, evaluation protocol substantially affects a published 500B-token conversion's scores in both directions across tasks, while its AR parent's scores vary by less than three points, so comparing dLLMs needs a common protocol. These results show that, in the tested low-budget regime, the frozen-tower configuration retains substantially more of the parent's generation performance than in-place conversion.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 28 of its 29 references that are in the map. First posted on arXiv 2 Oct 2026.

Builds on

28 of 29 references.

Closest ancestors · 15

Same direction, newest first

  1. 2026Nemotron-Labs-TwoTower: Diffusion Language Modeling with Pretrained Autoregressive Context25 Jun
  2. 2026Don't Retrain, Align: Adapting Autoregressive LMs to Diffusion LMs via Representation Alignment7 May
  3. 2026From AR to Diffusion: Efficiently Adapting Large Language Models with Strictly Causal and Elastic Horizons11 Apr
  4. 2026DreamOn: Diffusion Language Models For Code Infilling Beyond Fixed-size Canvas1 Feb
  5. 2025LLaDA2.0: Scaling Up Diffusion Language Models to 100B10 Dec
  6. 2025LLaDA-MoE: A Sparse MoE Diffusion Language Model29 Sep
  7. 2025Dream 7B: Diffusion Large Language Models21 Aug
  8. 2025The Diffusion Duality12 Jun
  9. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models12 Mar
  10. 2025Generalized Interpolating Discrete Diffusion6 Mar
  11. 2025Large Language Diffusion Models14 Feb
  12. 2024Beyond Autoregression: Fast LLMs via Self-Distillation Through Time28 Oct
3 more
  1. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Models23 Oct
  2. 2024Simple and Effective Masked Diffusion Language Models11 Jun
  3. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution25 Oct

Other references · 13

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  4. 2021Program Synthesis with Large Language Modelscited by 3,637
  5. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  6. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  7. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
  8. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  9. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
  10. 2024Discrete Flow Matchingcited by 317
  11. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  12. 2025Fast-dLLM v2: Efficient Block-Diffusion LLMcited by 139
1 more
  1. 2024Flow Matching with General Discrete Paths: A Kinetic-Optimal Perspectivecited by 57

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025From Next-Token to Next-Block: A Principled Adaptation Path for Diffusion LLMs
  2. 2026DreamingGoose: Staged Distillation from Autoregressive Transformers to Bidirectional Recurrent Diffusion Language Models
  3. 2026Enhancing Diffusion Language Models with Autoregressive Post-Training Weights
  4. 2026A Comparative analysis of Layer-wise Representational Capacity in AR and Diffusion LLMs
  5. 2025Efficient-DLM: From Autoregressive to Diffusion Language Models, and Beyond in Speed
  6. 2026Unlocking Lossless Speedups in LLMs via Discrete Diffusion
  7. 2025Esoteric Language Models: A Family of Any-Order Diffusion LLMs
  8. 2025Diffusion Language Models are Super Data Learners