mlsanityData to 14 Oct 2026

LightningRL: Breaking the Accuracy-Parallelism Trade-off of Block-wise dLLMs via Reinforcement Learning

arXiv 2603.13319 ↗First posted 4 Mar 2026cs.LGcited by 6

On the map: Deep reinforcement learning methods → RL training of language and agent models

Abstract (arXiv)

Diffusion Large Language Models (dLLMs) have emerged as a promising paradigm for parallel token generation, with block-wise variants garnering significant research interest. Despite their potential, existing dLLMs typically suffer from a rigid accuracy-parallelism trade-off: increasing the number of tokens per forward (TPF) via aggressive parallel decoding often leads to performance degradation and increased generation instability. We identify that this limitation stems from the model's inability to navigate high-parallelism regimes where approximation errors and local corruptions accumulate, ultimately undermining the reliability of parallel generation. To address this, we propose LightningRL, a post-training framework designed to directly optimize the speed-quality Pareto frontier of pre-trained dLLMs. Instead of forcing uniform parallelization, our approach leverages reinforcement learning to identify and reinforce high-parallelism trajectories that maintain generation accuracy. Built upon the Group Relative Policy Optimization (GRPO) framework, LightningRL introduces several enhancements tailored for dLLMs: (1) stabilized training via per-reward decoupled normalization; (2) token-level negative log-likelihood (NLL) regularization on correct trajectories to anchor model performance; and (3) a dynamic sampling strategy with TPF-aware filtering to enhance training efficiency. Experimental results across mathematical and coding benchmarks demonstrate that LightningRL consistently advances the Pareto frontier, achieving competitive task accuracy while significantly increasing parallelism, reaching an average TPF of 7.32 (with a peak of 11.10 on the MBPP dataset). Our code is available at https://github.com/SJTU-DENG-Lab/LightningRL.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods. Builds on 27 of its 30 references that are in the map. First posted on arXiv 4 Mar 2026.

Builds on

27 of 30 references.

References · 27

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2021Program Synthesis with Large Language Modelscited by 3,637
  6. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  7. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,027
  8. 2025Dream 7B: Diffusion Large Language Modelscited by 544
  9. 2022DiffuSeq: Sequence to Sequence Text Generation with Diffusion Modelscited by 450
  10. 2025Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decodingcited by 420
  11. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Modelscited by 409
  12. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Testcited by 331
15 more
  1. 2025DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generationcited by 187
  2. 2025d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learningcited by 186
  3. 2023Likelihood-Based Diffusion Language Modelscited by 151
  4. 2025Fast-dLLM v2: Efficient Block-Diffusion LLMcited by 139
  5. 2025SDAR: A Synergistic Diffusion-AutoRegression Paradigm for Scalable Sequence Generationcited by 129
  6. 2025Diffusion LLMs Can Do Faster-Than-AR Inference via Discrete Diffusion Forcingcited by 116
  7. 2025Revolutionizing Reinforcement Learning Framework for Diffusion Large Language Modelscited by 88
  8. 2024CLLMs: Consistency Large Language Modelscited by 72
  9. 2025dParallel: Learnable Parallel Decoding for dLLMscited by 65
  10. 2026d3LLM: Ultra-Fast Diffusion LLM using Pseudo-Trajectory Distillationcited by 37
  11. 2025Principled RL for Diffusion LLMs Emerges from a Sequence-Level Perspectivecited by 34
  12. 2025Learning to Parallel: Accelerating Diffusion Large Language Models via Learnable Parallel Decodingcited by 27
  13. 2024DiffPO: A causal diffusion model for learning distributions of potential outcomescited by 17
  14. 2025LoPA: Scaling dLLM Inference via Lookahead Parallel Decodingcited by 17
  15. 2025Co-GRPO: Co-Optimized Group Relative Policy Optimization for Masked Diffusion Modelcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026DMax: Aggressive Parallel Decoding for dLLMscited by 18
  2. 2026Introspective Diffusion Language Modelscited by 8
  3. 2026Multi-Block Diffusion Language Modelscited by 3
  4. 2026TAD: Temporal-Aware Trajectory Self-Distillation for Fast and Accurate Diffusion LLMcited by 2
  5. 2026SLIM-RL: Risk-Budgeted Random-Masking RL for Diffusion LLMs Without Trajectory Slicingcited by 0
  6. 2026Faster Block-Diffusion Serving with Distribution-Free Risk Guaranteescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025d-TreeRPO: Towards More Reliable Policy Optimization for Diffusion Language Models
  2. 2026Accelerating Masked Diffusion Large Language Models: A Survey of Efficient Inference Techniques
  3. 2026DARE: Diffusion Large Language Models Alignment and Reinforcement Executor
  4. 2025PipelineRL: Faster On-policy Reinforcement Learning for Long Sequence Generation
  5. 2025dUltra: Ultra-Fast Diffusion Language Models via Reinforcement Learning
  6. 2026dTRPO: Trajectory Reduction in Policy Optimization of Diffusion Large Language Models
  7. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning
  8. 2025TritonRL: Training LLMs to Think and Code Triton Without Cheating