mlsanityData to 14 Oct 2026

Consolidating Reinforcement Learning for Multimodal Discrete Diffusion Models

arXiv 2510.02880 ↗First posted 3 Oct 2025cs.AIcited by 12

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 2 more
Abstract (arXiv)

Optimizing discrete diffusion model (DDM) with rewards remains a challenge: the non-autoregressive paradigm makes importance sampling intractable and rollout complex, puzzling reinforcement learning methods such as Group Relative Policy Optimization (GRPO). In this study, we introduce MaskGRPO, the first viable approach to enable scalable multimodal reinforcement learning in discrete diffusion with effective importance sampling and modality-specific adaptations. To this end, we first clarify the theoretical foundation for DDMs, which facilitates building an importance estimator that captures valuable token fluctuation for gradient updates. We then delicately tailored the rollout method for visual sequences, which yields diverse completions and reliable optimization gradients. Upon math reasoning, coding, and visual generation benchmarks, MaskGRPO brings more stable and efficient updates, leading to stronger reasoning performance and better generation quality. This study establishes MaskGRPO as a systematic policy optimization approach and the first practical way for discretized visual diffusion.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 33 of its 36 references that are in the map. First posted on arXiv 3 Oct 2025.

Builds on

33 of 36 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  2. 2025Flow-GRPO: Training Flow Matching Models via Online RL8 May
  3. 2023Let's Verify Step by Step31 May
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  5. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 28

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  4. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  5. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  6. 2022MaskGIT: Masked Generative Image Transformercited by 1,099
  7. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
  8. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 972
  9. 2025Large Language Diffusion Modelscited by 912
  10. 2024Simple and Effective Masked Diffusion Language Modelscited by 848
  11. 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 812
  12. 2024Emu3: Next-Token Prediction is All You Needcited by 781
16 more
  1. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
  2. 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignmentcited by 475
  3. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Modelscited by 409
  4. 2025BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Datasetcited by 376
  5. 2025LLaDA 1.5: Variance-Reduced Preference Optimization for Large Language Diffusion Modelscited by 285
  6. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 250
  7. 2025HPSv3: Towards Wide-Spectrum Human Preference Scorecited by 211
  8. 2024Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Samplingcited by 208
  9. 2025Unified Reward Model for Multimodal Understanding and Generationcited by 199
  10. 2025DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generationcited by 187
  11. 2025d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learningcited by 186
  12. 2025Teaching Large Language Models to Regress Accurate Image Quality Scores using Score Distributioncited by 141
  13. 2025X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Againcited by 112
  14. 2025wd1: Weighted Policy Optimization for Reasoning in Diffusion Language Modelscited by 71
  15. 2024Analyzing The Language of Visual Tokenscited by 12
  16. 2025ReDDiT: Rehashing Noise for Discrete Visual Generationcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 11.

12 papers.

  1. 2026Back on Track: Aligning Rewards and States for Reasoning in Diffusion Large Language Modelscited by 4
  2. 2025Reasoning in Diffusion Large Language Models is Concentrated in Dynamic Confusion Zonescited by 3
  3. 2026Advances in GRPO for Generation Models: A Surveycited by 2
  4. 2026AceTone: Bridging Words and Colors for Conditional Image Gradingcited by 2
  5. 2026dFlowGRPO: Rate-Aware Policy Optimization for Discrete Flow Modelscited by 2
  6. 2026Sketch Then Paint: Hierarchical Reinforcement Learning for Diffusion Multi-Modal Large Language Modelscited by 1
  7. 2026Efficient Reinforcement for Visual-Textual Thinking with Discrete Diffusion Modelcited by 0
  8. 2026JAGG: Jacobian-Aggregated Group Gradient for Efficient GRPO Training of Diffusion Modelscited by 0
  9. 2026CT-OPD: Counterfactual Trace On-Policy Distillation for Diffusion Vision-Language Modelscited by 0
  10. 2026Reinforcement Learning on the Discrete Composition Channel of a Crystal Generator: Validated Gains and Reward Hackingcited by 0
  11. 2026Towards Unbiased On-Policy Distillation for Block Diffusion Language Modelscited by 0
  12. 2026MaskSense: Confronting the Visual Exploration Trap in Masked Image Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026UDM-GRPO: Stable and Efficient Group Relative Policy Optimization for Uniform Discrete Diffusion Models
  2. 2026A Continuous-Time Reinforcement Learning Framework for Fine-Tuning Discrete Diffusion Models
  3. 2025MDPO: Overcoming the Training-Inference Divide of Masked Diffusion Language Models
  4. 2023Training Diffusion Models with Reinforcement Learning
  5. 2025Reinforcement Learning Meets Masked Generative Models: Mask-GRPO for Text-to-Image Generation
  6. 2024Aligning Few-Step Diffusion Models with Dense Reward Difference Learning
  7. 2026MAR-GRPO: Stabilized GRPO for AR-diffusion Hybrid Image Generation
  8. 2025Co-GRPO: Co-Optimized Group Relative Policy Optimization for Masked Diffusion Model