mlsanityData to 7 Oct 2026

dQwen3.5: Hybrid-Attention Diffusion Language Models

arXiv 2609.20751 ↗First posted 17 Sep 2026cs.CL cs.LGcited by 0

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Adapting a pretrained autoregressive (AR) model is a cost-efficient route to a diffusion language model (DLM). While nearly all such adaptations start from a full-attention transformer, AR modeling has shifted toward hybrid architectures that interleave attention and RNN layers. This creates an obstacle for adaptation: unlike attention, RNNs are structurally causal and nontrivial to bidirectionalize. Despite this mismatch, we investigate whether such backbones can become effective DLMs by adapting Qwen3.5 at 0.8B, 2B, 4B, and 9B scales, yielding the dQwen3.5 family. We find that hybrid backbones can be efficient starting points for adaptation: against a full-attention control, the hybrid reaches a given training loss in about half the tokens. Across scales, dQwen3.5 resembles full-attention DLMs in any-order decoding behavior and performs strongly under parallel decoding.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 42 of its 44 references that are in the map. First posted on arXiv 17 Sep 2026.

Builds on

42 of 44 references.

Closest ancestors · 16

Same direction, newest first

  1. 2026Improved Large Language Diffusion Models24 Jun
  2. 2025Efficient-DLM: From Autoregressive to Diffusion Language Models, and Beyond in Speed16 Dec
  3. 2025LLaDA2.0: Scaling Up Diffusion Language Models to 100B10 Dec
  4. 2025SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generation7 Oct
  5. 2025CoDA: Coding LM via Diffusion Adaptation27 Sep
  6. 2025Dream-Coder 7B: An Open Diffusion Language Model for Code1 Sep
  7. 2025Dream 7B: Diffusion Large Language Models21 Aug
  8. 2025DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generation25 Jun
  9. 2025Anchored Diffusion Language Model24 May
  10. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models12 Mar
  11. 2025Large Language Diffusion Models14 Feb
  12. 2024Scaling up Masked Diffusion Models on Text24 Oct
4 more
  1. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Models23 Oct
  2. 2024Simple and Effective Masked Diffusion Language Models11 Jun
  3. 2024Simplified and Generalized Masked Diffusion for Discrete Data6 Jun
  4. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution25 Oct

Other references · 26

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2024Qwen2.5 Technical Reportcited by 4,713
  7. 2023Let's Verify Step by Stepcited by 3,598
  8. 2021Program Synthesis with Large Language Modelscited by 3,509
  9. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  10. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  11. 2024Qwen2.5-Coder Technical Reportcited by 1,151
  12. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
14 more
  1. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  2. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 390
  3. 2025Mercury: Ultra-Fast Language Models Based on Diffusioncited by 225
  4. 2024Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Datasetcited by 153
  5. 2023Continual Pre-Training of Large Language Models: How to (re)warm your model?cited by 135
  6. 2018Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Contextcited by 113
  7. 2025NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Modelcited by 94
  8. 2021Do Long-Range Language Models Actually Use Long-Range Context?cited by 92
  9. 2025Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 85
  10. 2021What Context Features Can Transformer Language Models Use?cited by 49
  11. 2025Nemotron-CC-Math: A 133 Billion-Token-Scale High Quality Math Pretraining Datasetcited by 34
  12. 2026Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decodingcited by 19
  13. 2025Demystifying Diffusion Objectives: Reweighted Losses are Better Variational Boundscited by 7
  14. 2026FLARE: Diffusion for Hybrid Language Modelcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Retrofitting Linear Attention into Diffusion Language Models
  2. 2026PreDiff-LM: Pretrained Discrete Masked Diffusion Language Modeling with Hybrid Attention
  3. 2025From Next-Token to Next-Block: A Principled Adaptation Path for Diffusion LLMs
  4. 2025Attention Sinks in Diffusion Language Models
  5. 2025DiffusionVL: Translating Any Autoregressive Models into Diffusion Vision Language Models
  6. 2026Enhancing Diffusion Language Models with Autoregressive Post-Training Weights
  7. 2026Don't Retrain, Align: Adapting Autoregressive LMs to Diffusion LMs via Representation Alignment
  8. 2025Discrete Diffusion in Large Language and Multimodal Models: A Survey