mlsanityData to 7 Oct 2026

Entropy Aware Reward Guidance for Diffusion Language Model Alignment

arXiv 2602.05000 ↗First posted 4 Feb 2026cs.LG cs.AI cs.CLcited by 2

On the map: Diffusion models → Discrete and text diffusion

also in 1 more
Abstract (arXiv)

Reward guidance, also known as posterior sampling, is a popular method for test-time adaptation and post-training in continuous diffusion models. In this paper, we study reward guidance for discrete diffusion language models; now, one cannot differentiate through the natural outputs of the model because they are discrete tokens. We introduce a novel mechanism called EntRGi (Entropy aware Reward Guidance) to address this issue. EntRGi dynamically interpolates between continuous token relaxations and sampled hard tokens, on a token-by-token basis, using the diffusion model's predictive entropy. We demonstrate that EntRGi maintains both reward model reliability and optimization accuracy, while existing approaches sacrifice one for the other. We empirically validate our approach on 7B-parameter diffusion language models across two settings: (1) test-time adaptation, and (2) RGRL (Reward Guided Reinforcement Learning), our recipe for post-training on reward-guided data, showing consistent improvements over state-of-the-art methods. Our code is available at https://atutej.github.io/entrgi-rgrl

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 56 of its 62 references that are in the map. First posted on arXiv 4 Feb 2026.

Builds on

56 of 62 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025Test-Time Anchoring for Discrete Diffusion Posterior Sampling2 Oct
  2. 2025Dream-Coder 7B: An Open Diffusion Language Model for Code1 Sep
  3. 2025Dream 7B: Diffusion Large Language Models21 Aug
  4. 2025TESS 2: A Large-Scale Generalist Diffusion Language Model19 Feb
  5. 2025Large Language Diffusion Models14 Feb
  6. 2024Simple and Effective Masked Diffusion Language Models11 Jun
  7. 2024Simplified and Generalized Masked Diffusion for Discrete Data6 Jun
  8. 2023Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution25 Oct

Other references · 48

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  4. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  5. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  6. 2021Structured Denoising Diffusion Models in Discrete State-Spacescited by 1,665
  7. 2022Diffusion Posterior Sampling for General Noisy Inverse Problemscited by 1,375
  8. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  9. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  10. 2023Training Diffusion Models with Reinforcement Learningcited by 939
  11. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  12. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
36 more
  1. 2023DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Modelscited by 431
  2. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewardscited by 431
  3. 2023Universal Guidance for Diffusion Modelscited by 426
  4. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
  5. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 341
  6. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 294
  7. 2023FreeDoM: Training-Free Energy-Guided Conditional Diffusion Modelcited by 290
  8. 2023Style Aligned Image Generation via Shared Attentioncited by 214
  9. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 204
  10. 2025A General Framework for Inference-time Scaling and Steering of Diffusion Modelscited by 201
  11. 2025d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learningcited by 178
  12. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Stylecited by 168
  13. 2024Semantic Image Inversion and Editing using Rectified Stochastic Differential Equationscited by 162
  14. 2023Manifold Preserving Guided Diffusioncited by 153
  15. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 136
  16. 2024Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-Based Decodingcited by 127
  17. 2024TFG: Unified Training-Free Guidance for Diffusion Modelscited by 103
  18. 2024Video Diffusion Alignment via Reward Gradientscited by 101
  19. 2023Confronting Reward Model Overoptimization with Constrained RLHFcited by 96
  20. 2025Test-time Alignment of Diffusion Models without Reward Over-optimizationcited by 89
  21. 2023Beyond First-Order Tweedie: Solving Inverse Problems using Latent Diffusioncited by 78
  22. 2024Fine-Tuning Discrete Diffusion Models via Reward Optimization with Applications to DNA and Protein Designcited by 72
  23. 2023Prompt-tuning latent diffusion models for inverse problemscited by 70
  24. 2025Inference-time Scaling of Diffusion Models through Classical Searchcited by 48
  25. 2025Fine-Tuning Discrete Diffusion Models with Policy Gradient Methodscited by 36
  26. 2024LMUnit: Fine-grained Evaluation with Natural Language Unit Testscited by 32
  27. 2025Diffusion Tree Sampling: Scalable inference-time alignment of diffusion modelscited by 31
  28. 2024Steering Masked Discrete Diffusion Models via Discrete Denoising Posterior Predictioncited by 28
  29. 2025Test-Time Scaling of Diffusion Models via Noise Trajectory Searchcited by 28
  30. 2025Training-Free Guidance Beyond Differentiability: Scalable Path Steering with Tree Search in Diffusion and Flow Modelscited by 23
  31. 2025Language Models that Think, Chat Bettercited by 23
  32. 2025Split Gibbs Discrete Diffusion Posterior Samplingcited by 18
  33. 2025TR2-D2: Tree Search Guided Trajectory-Aware Fine-Tuning for Discrete Diffusioncited by 13
  34. 2025Preference-Based Alignment of Discrete Diffusion Modelscited by 12
  35. 2025Inference-Time Scaling of Discrete Diffusion Models via Importance Weighting and Optimal Proposal Designcited by 12
  36. 2025Fine-Tuning Diffusion Models via Intermediate Distribution Shapingcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026VGAS: Variance-Reduced Guidance and Adaptive Selection for Training-Free Reward Alignment in Discrete Diffusioncited by 0
  2. 2026FastGuide: Accelerating Reward Guidance for Diffusion Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Mode-Seeking RL: Trajectory-Balance Post-Training for Diffusion Language Models
  2. 2025Advancing Reasoning in Diffusion Language Models with Denoising Process Rewards
  3. 2026Reinforcement Learning for Diffusion LLMs with Entropy-Guided Step Selection and Stepwise Advantages
  4. 2024Aligning Diffusion Behaviors with Q-functions for Efficient Continuous Control
  5. 2026Explore or Converge? Stage-Guided Per-Step Optimization for Diffusion Models
  6. 2026Back on Track: Aligning Rewards and States for Reasoning in Diffusion Large Language Models
  7. 2025GIFT: Guided Importance-Aware Fine-Tuning for Diffusion Language Models
  8. 2026Breaking the Factorization Barrier in Diffusion Language Models