# Sample-Adaptive Latent Rewards for Uncertainty-Guided Diffusion Post-Training

arXiv: https://arxiv.org/abs/2608.06125 · HTML: https://mlsanity.com/paper/2608.06125/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models.
- Builds on 16 of its 25 references that are in the map.
- First posted on arXiv 6 Aug 2026.

## Placed in

- Diffusion models / [Diffusion and flow policies for robotics](https://mlsanity.com/diffusion/diffusion-and-flow-policies-for-robotics/)
- LLM alignment / RLHF / preference optimisation / [Diffusion and multimodal model alignment](https://mlsanity.com/alignment/diffusion-and-multimodal-model-alignment/)

## Builds on (most cited first, 16 of 16)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023)
- [Flow-GRPO: Training Flow Matching Models via Online RL](https://mlsanity.com/paper/2505.05470/) (2025)
- [Aligning Text-to-Image Diffusion Models with Reward Backpropagation](https://mlsanity.com/paper/2310.03739/) (2023)
- [Improving Video Generation with Human Feedback](https://mlsanity.com/paper/2501.13918/) (2025)
- [HPSv3: Towards Wide-Spectrum Human Preference Score](https://mlsanity.com/paper/2508.03789/) (2025)
- [VisionReward: Fine-Grained Multi-Dimensional Human Preference Learning for Image and Video Generation](https://mlsanity.com/paper/2412.21059/) (2024)
- [VideoDPO: Omni-Preference Alignment for Video Diffusion Generation](https://mlsanity.com/paper/2412.14167/) (2024)
- [Unified Multimodal Chain-of-Thought Reward Model through Reinforcement Fine-Tuning](https://mlsanity.com/paper/2505.03318/) (2025)
- [GenAI Arena: An Open Evaluation Platform for Generative Models](https://mlsanity.com/paper/2406.04485/) (2024)
- [Diffusion Model as a Noise-Aware Latent Reward Model for Step-Level Preference Optimization](https://mlsanity.com/paper/2502.01051/) (2025)
- [VQ-Insight: Teaching VLMs for AI-Generated Video Quality Understanding via Progressive Visual Reinforcement Learning](https://mlsanity.com/paper/2506.18564/) (2025)
- [Video Generation Models Are Good Latent Reward Models](https://mlsanity.com/paper/2511.21541/) (2025)
- [Seeing What Matters: Visual Preference Policy Optimization for Visual Generation](https://mlsanity.com/paper/2511.18719/) (2025)
- [DRM: Diffusion-based Reward Model With Step-wise Guidance](https://mlsanity.com/paper/2605.25661/) (2026)

## Abstract (arXiv)

Latent reward models can supervise visual diffusion models without decoding intermediate states into pixel space. This makes alignment with human preferences more efficient. However, existing latent reward models output only scalar scores. They do not estimate the uncertainty of each prediction. The generator therefore cannot determine which feedback is reliable. This can drive optimization in the wrong direction and lead to reward hacking. We propose textscSURE, a unified latent-space framework for image and video diffusion models. It learns reward distributions and directly uses their reliability to guide dense post-training. First, we propose sample-adaptive latent reward model (textscSURE-LRM). It predicts a Gaussian utility for each noisy latent. Its mean predicts the reward score. Its variance reflect the uncertainty of prediction without human annotation. The learned distribution then guides post-training through uncertainty-guided reward feedback learning (textscSURE-REFL). This method provides uncertainty-guided dense feedback along the denoising trajectory. At selected transitions, textscSURE-REFL queries the frozen textscSURE-LRM. It converts detached variance into reliability weights for samples at the same transition. Each weighted reward is backpropagated only through its local transition. The entire process remains in latent space and requires neither pixel-space decoding nor the full denoising graph. Experiments show that textscSURE-LRM improves preference prediction over strong baselines. textscSURE-REFL achieves the sota performance among various metrics and further improves optimization stability. It also achieves the highest VBench quality, semantic, and total scores among the evaluated methods.