mlsanityData to 14 Oct 2026

AR-GRPO: Training Autoregressive Image Generation Models via Reinforcement Learning

arXiv 2508.06924 ↗First posted 9 Aug 2025cs.CVcited by 22

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Inspired by the success of reinforcement learning (RL) in refining large language models (LLMs), we propose AR-GRPO, an approach to integrate online RL training into autoregressive (AR) image generation models. We adapt the Group Relative Policy Optimization (GRPO) algorithm to refine the vanilla autoregressive models' outputs by carefully designed reward functions that evaluate generated images across multiple quality dimensions, including perceptual quality, realism, and semantic fidelity. We conduct comprehensive experiments on both class-conditional (i.e., class-to-image) and text-conditional (i.e., text-to-image) image generation tasks, demonstrating that our RL-enhanced framework significantly improves both the image quality and human preference of generated images compared to the standard AR baselines. Our results show consistent improvements across various evaluation metrics, establishing the viability of RL-based optimization for AR image generation and opening new avenues for controllable and high-quality image synthesis. The source codes and models are available at: https://github.com/Kwai-Klear/AR-GRPO.

Key facts. Cited by 22 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 61 of its 72 references that are in the map. First posted on arXiv 9 Aug 2025.

Builds on

61 of 72 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025Reasoning with Exploration: An Entropy Perspective17 Jun
  2. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning2 Jun
  3. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  4. 2025DanceGRPO: Unleashing GRPO on Visual Generation12 May
  5. 2025Flow-GRPO: Training Flow Matching Models via Online RL8 May
  6. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  7. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan

Other references · 54

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2022Training language models to follow instructions with human feedbackcited by 15,971
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2020Denoising Diffusion Implicit Modelscited by 9,781
  8. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  9. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  10. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  11. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  12. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
38 more
  1. 2022Scalable Diffusion Models with Transformerscited by 6,054
  2. 2022Classifier-Free Diffusion Guidancecited by 5,847
  3. 2017Neural Discrete Representation Learningcited by 4,712
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  6. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  7. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  8. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,588
  9. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  10. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  11. 2016Improved Techniques for Training GANscited by 2,970
  12. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,626
  13. 2020GLU Variants Improve Transformercited by 1,804
  14. 2019Root Mean Square Layer Normalizationcited by 1,480
  15. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,387
  16. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  17. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,045
  18. 2017StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translationcited by 1,039
  19. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,023
  20. 2023Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generationcited by 977
  21. 2023GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignmentcited by 972
  22. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 944
  23. 2023Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesiscited by 890
  24. 2016Pixel Recurrent Neural Networkscited by 816
  25. 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 812
  26. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 777
  27. 2022Autoregressive Image Generation using Residual Quantizationcited by 722
  28. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 694
  29. 2021Vector-quantized Image Modeling with Improved VQGANcited by 658
  30. 2022MANIQA: Multi-dimension Attention Network for No-Reference Image Quality Assessmentcited by 522
  31. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 457
  32. 2021Generating Images with Sparse Representationscited by 266
  33. 2018A Note on the Inception Scorecited by 226
  34. 2025Unified Reward Model for Multimodal Understanding and Generationcited by 199
  35. 2025T2I-R1: Reinforcing Image Generation with Collaborative Semantic-level and Token-level CoTcited by 166
  36. 2024JanusFlow: Harmonizing Autoregression and Rectified Flow for Unified Multimodal Understanding and Generationcited by 163
  37. 2025Teaching Large Language Models to Regress Accurate Image Quality Scores using Score Distributioncited by 141
  38. 2017PixelSNAIL: An Improved Autoregressive Generative Modelcited by 137

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 10; 2026: 12.

22 papers.

Sort:

By area:

  1. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  2. 2026Flow-OPD: On-Policy Distillation for Flow Matching Modelscited by 34
  3. 2026JavisDiT++: Unified Modeling and Optimization for Joint Audio-Video Generationcited by 25
  4. 2025Group Critical-token Policy Optimization for Autoregressive Image Generationcited by 17
  5. 2025STAGE: Stable and Generalizable GRPO for Autoregressive Image Generationcited by 15
  6. 2026NextFlow: Unified Sequential Modeling Activates Multimodal Understanding and Generationcited by 14
  7. 2025Unified Multimodal Models as Auto-Encoderscited by 13
  8. 2025COOPER: A Unified Model for Cooperative Perception and Reasoning in Spatial Intelligencecited by 9
  9. 2026Understanding Reward Hacking in Text-to-Image Reinforcement Learningcited by 9
  10. 2025RubricRL: Simple Generalizable Rewards for Text-to-Image Generationcited by 7
  11. 2025VA-π: Variational Policy Alignment for Pixel-Aware Autoregressive Generationcited by 7
  12. 2025IRG-MotionLLM: Interleaving Motion Generation, Assessment and Refinement for Text-to-Motion Generationcited by 6
10 more
  1. 2025Expand and Prune: Maximizing Trajectory Diversity for Effective GRPO in Generative Modelscited by 6
  2. 2026ConsistentRFT: Reducing Visual Hallucinations in Flow-based Reinforcement Fine-Tuningcited by 6
  3. 2026Power Reinforcement Post-Training of Text-to-Image Models with Super-Linear Advantage Shapingcited by 3
  4. 2026VAR RL Done Right: Tackling Asynchronous Policy Conflicts in Visual Autoregressive Generationcited by 2
  5. 2026Advances in GRPO for Generation Models: A Surveycited by 2
  6. 2026Sketch Then Paint: Hierarchical Reinforcement Learning for Diffusion Multi-Modal Large Language Modelscited by 1
  7. 2025Enhancing Blind Face Restoration through Online Reinforcement Learningcited by 0
  8. 2026Policy-based Tuning of Autoregressive Image Models with Instance- and Distribution-Level Rewardscited by 0
  9. 2026UniVR: Thinking in Visual Space for Unified Visual Reasoningcited by 0
  10. 2026JAGG: Jacobian-Aggregated Group Gradient for Efficient GRPO Training of Diffusion Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Delving into RL for Image Generation with CoT: A Study on DPO vs. GRPO
  2. 2025Reinforcement Learning Meets Masked Generative Models: Mask-GRPO for Text-to-Image Generation
  3. 2025Fine-Tuning Next-Scale Visual Autoregressive Models with Group Relative Policy Optimization
  4. 2025X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Again
  5. 2025Integrating Reinforcement Learning with Visual Generative Models: Foundations and Advances
  6. 2026Balancing Performance and Diversity in GRPO Autoregressive Text-to-Image Post-Training
  7. 2026UniGRPO: Unified Policy Optimization for Reasoning-Driven Visual Generation
  8. 2026From Broad Exploration to Stable Synthesis: Entropy-Guided Optimization for Autoregressive Image Generation