mlsanityData to 14 Oct 2026

CanvasAnneal: Curriculum Reinforcement Learning for Diffusion Language Models

arXiv 2609.13060 ↗First posted 11 Sep 2026cs.LGcited by 0

On the map: LLM reasoning → Diffusion reasoning and test-time scaling

also in 1 more
Abstract (arXiv)

Diffusion Language Models (DLMs) offer promising parallel generation capabilities but lag behind autoregressive models in complex reasoning and tool-use tasks. While Reinforcement Learning (RL) has recently been applied to enhance DLMs, standard RL approaches suffer from an exploration bottleneck. To address this, we inject reasoning priors from a stronger teacher model to guide RL exploration. In this paper, we introduce CanvasAnneal, a curriculum-guided diffusion RL framework. During the initial RL phase, we warm-start exploration by injecting teacher-generated reasoning traces into the initial diffusion canvas. As training progresses, we gradually remove this guidance and require the model to generate more of the reasoning trajectory independently. Across mathematical reasoning and tool-use benchmarks, CanvasAnneal improves over standard diffu-GRPO on MATH500, Countdown, and Tau2 and substantially accelerates reward improvement on several tasks, while gains are task-dependent. Our results suggest that structured training-time guidance can alleviate exploration bottlenecks in diffusion RL and speed up convergence on harder tasks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Diffusion reasoning and test-time scaling, one of 17 directions in LLM reasoning. Builds on 27 of its 33 references that are in the map. First posted on arXiv 11 Sep 2026.

Builds on

27 of 33 references.

Closest ancestors · 8

Same direction, newest first

  1. 2026Mask-Aware Policy Gradients for Diffusion Language Models16 Jul
  2. 2026How Transparent is DiffusionGemma?18 Jun
  3. 2026Learnability-Informed Fine-Tuning of Diffusion Language Models21 May
  4. 2025Simple Policy Gradients for Reasoning with Diffusion Language Models5 Oct
  5. 2025d2: Improving Reasoning in Diffusion Language Models via Trajectory Likelihood Estimation25 Sep
  6. 2025wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Models7 Jul
  7. 2025Reinforcing the Diffusion Chain of Lateral Thought with Diffusion Language Models15 May
  8. 2025d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning16 Apr

Other references · 19

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2025Large Language Diffusion Modelscited by 912
  5. 2024Simple and Effective Masked Diffusion Language Modelscited by 848
  6. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 803
  7. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distributioncited by 745
  8. 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 627
  9. 2025Dream 7B: Diffusion Large Language Modelscited by 544
  10. 2025Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM Reasoningcited by 92
  11. 2025BREAD: Branched Rollouts from Expert Anchors Bridge SFT & RL for Reasoningcited by 63
  12. 2026d3LLM: Ultra-Fast Diffusion LLM using Pseudo-Trajectory Distillationcited by 37
7 more
  1. 2025Inpainting-Guided Policy Optimization for Diffusion Large Language Modelscited by 16
  2. 2026Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillationcited by 11
  3. 2026Stabilizing Reinforcement Learning for Diffusion Language Modelscited by 10
  4. 2026Lookahead-then-Verify: Reliable Constrained Decoding for Diffusion LLMs under Context-Free Grammarscited by 7
  5. 2026The Bitter Lesson of Diffusion Language Models for Agentic Workflows: A Comprehensive Reality Checkcited by 6
  6. 2026GDSD: Reinforcement Learning as Guided Denoiser Self-Distillation for Diffusion Language Modelscited by 6
  7. 2026TRIMS: Trajectory-Ranked Instruction Masked Supervision for Diffusion Language Modelscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Revolutionizing Reinforcement Learning Framework for Diffusion Large Language Models
  2. 2026Test-Time Scaling with Diffusion Language Models via Reward-Guided Stitching
  3. 2025DiRL: An Efficient Post-Training Framework for Diffusion Language Models
  4. 2025Advancing Reasoning in Diffusion Language Models with Denoising Process Rewards
  5. 2026Curriculum Reinforcement Learning Can Incentivize Reasoning Capacity in LLMs Beyond the Base Model
  6. 2026LaViDa-R1: Advancing Reasoning for Unified Multimodal Diffusion Language Models
  7. 2025RFG: Test-Time Scaling for Diffusion Large Language Model Reasoning with Reward-Free Guidance
  8. 2025Reasoning in Diffusion Large Language Models is Concentrated in Dynamic Confusion Zones