mlsanityData to 14 Oct 2026

Balancing Performance and Diversity in GRPO Autoregressive Text-to-Image Post-Training

arXiv 2606.21498 ↗First posted 19 Jun 2026cs.AI cs.LGcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Autoregressive text-to-image (T2I) generation has recently advanced rapidly, yet aligning generated images with human preferences remains challenging. GRPO-style online reinforcement learning provides an effective framework; however, existing methods typically treat reference-policy divergence as fixed, despite its direct impact on policy optimization. We study this overlooked factor within a unified f-divergence framework, encompassing forward KL, reverse KL, and JS divergence, for GRPO-style autoregressive T2I alignment. Our systematic theoretical analysis reveals that different divergences reshape token-level updates in distinct ways. In particular, under the sampled-token shaping form used, JS regularization achieves a favorable trade-off by mitigating uniform bias relative to the reference policy while still discouraging large deviations. Extensive experiments on LlamaGen and Janus-7B show that JS divergence achieves the strongest or highly competitive optimization performance on most evaluation metrics while maintaining favorable generation diversity. The code is available at https://github.com/tuoyou-hao/BPD-GRPO.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 14 of its 51 references that are in the map. First posted on arXiv 19 Jun 2026.

Builds on

14 of 51 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026Power Reinforcement Post-Training of Text-to-Image Models with Super-Linear Advantage Shaping11 May
  2. 2025Principled RL for Flow Matching Emerges from the Chunk-level Policy Optimization24 Oct
  3. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 11

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 890
  3. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 777
  4. 2024ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignmentcited by 475
  5. 2024GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generationcited by 135
  6. 2023Aligning Language Models with Preferences through f-divergence Minimizationcited by 120
  7. 2024Aesthetic Post-Training Diffusion Models from Generic Preferences with Step-by-step Preference Optimizationcited by 84
  8. 2017Policy Gradient Methods for Reinforcement Learning with Function Approximation and Action-Dependent Baselinescited by 20
  9. 2025Group Critical-token Policy Optimization for Autoregressive Image Generationcited by 17
  10. 2025Reinforcement Fine-Tuning Powers Reasoning Capability of Multimodal Large Language Modelscited by 14
  11. 2025The Hidden Link Between RLHF and Contrastive Learningcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025STAGE: Stable and Generalizable GRPO for Autoregressive Image Generation
  2. 2025AR-GRPO: Training Autoregressive Image Generation Models via Reinforcement Learning
  3. 2025Flow-GRPO: Training Flow Matching Models via Online RL
  4. 2025Reinforcement Learning Meets Masked Generative Models: Mask-GRPO for Text-to-Image Generation
  5. 2026MAR-GRPO: Stabilized GRPO for AR-diffusion Hybrid Image Generation
  6. 2026Policy-based Tuning of Autoregressive Image Models with Instance- and Distribution-Level Rewards
  7. 2026Advances in GRPO for Generation Models: A Survey
  8. 2026V-GRPO: Online Reinforcement Learning for Denoising Generative Models Is Easier than You Think