Mechanism Shift During Post-training from Autoregressive to Masked Diffusion Language Models
On the map: Diffusion models → Discrete and text diffusion
Abstract (arXiv)
Post-training pretrained autoregressive models (ARMs) into masked diffusion models (MDMs) provides an efficient route to diffusion language modeling, but it remains unclear whether the resulting models reuse inherited autoregressive computation or reorganize it for non-autoregressive generation. We compare two 7B ARM-MDM families across four controlled diagnostic tasks and find a task-dependent mechanism shift. On prefix-dominant tasks, MDMs largely preserve inherited high-attribution pathways or exhibit only modest changes in where computation occurs. On globally constrained tasks, the reorganization is substantially stronger, with task-relevant computation shifting toward earlier layers. This depth-wise pattern persists across prompt resampling, circuit budgets, and tested inference budgets, while targeted ablations support the functional importance of the identified structures under the tested intervention protocols. At the component level, diagnostic probes suggest that ARMs rely more strongly on sharply specialized components, whereas MDMs exhibit weaker single-component specialization and more diffuse output-space alignment. Together, these results suggest that diffusion post-training selectively preserves or reorganizes inherited computation according to task structure, rather than uniformly replacing autoregressive mechanisms.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 16 of its 25 references that are in the map. First posted on arXiv 21 Jan 2026.
Builds on
16 of 25 references.
Closest ancestors · 2
Same direction, newest first
- 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Models23 Oct
- 2024Beyond Autoregression: Discrete Diffusion for Complex Reasoning and Planning18 Oct
Other references · 14
- 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,719
- 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
- 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 803
- 2015Sequence Level Training with Recurrent Neural Networkscited by 576
- 2019Neural Text Generation with Unlikelihood Trainingcited by 503
- 2017Non-Autoregressive Neural Machine Translationcited by 463
- 2023Attribution Patching Outperforms Automated Circuit Discoverycited by 209
- 2023Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchillacited by 149
- 2024Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Trackingcited by 149
- 2024Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanismscited by 149
- 2024LLM Circuit Analyses Are Consistent Across Training and Scalecited by 63
- 2024Finding Transformer Circuits with Edge Pruningcited by 60
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026DreamingGoose: Staged Distillation from Autoregressive Transformers to Bidirectional Recurrent Diffusion Language Models
- 2026Multi-Stream Residual Computation in Diffusion Language Models
- 2026Don't Retrain, Align: Adapting Autoregressive LMs to Diffusion LMs via Representation Alignment
- 2025Activation Steering for Masked Diffusion Language Models
- 2026Reducing Pretraining-Generation Mismatch in Diffusion Language Models
- 2023Diffusion Language Models Can Perform Many Tasks with Scaling and Instruction-Finetuning
- 2026A Theoretical Analysis of Why Masked Diffusion Models Mitigate the Reversal Curse
- 2025Diffusion Beats Autoregressive in Data-Constrained Settings