# Group Critical-token Policy Optimization for Autoregressive Image Generation

arXiv: https://arxiv.org/abs/2509.22485 · HTML: https://mlsanity.com/paper/2509.22485/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 56 references that are in the map.
- First posted on arXiv 26 Sep 2025.

## Builds on (most cited first, 30 of 50)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [Kimi k1.5: Scaling Reinforcement Learning with LLMs](https://mlsanity.com/paper/2501.12599/) (2025)
- [ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation](https://mlsanity.com/paper/2304.05977/) (2023)
- [Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation](https://mlsanity.com/paper/2305.01569/) (2023)
- [GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment](https://mlsanity.com/paper/2310.11513/) (2023)
- [Chameleon: Mixed-Modal Early-Fusion Foundation Models](https://mlsanity.com/paper/2405.09818/) (2024)
- [Emerging Properties in Unified Multimodal Pretraining](https://mlsanity.com/paper/2505.14683/) (2025)
- [Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis](https://mlsanity.com/paper/2306.09341/) (2023)
- [Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling](https://mlsanity.com/paper/2501.17811/) (2025)
- [Emu3: Next-Token Prediction is All You Need](https://mlsanity.com/paper/2409.18869/) (2024)
- [Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation](https://mlsanity.com/paper/2406.06525/) (2024)
- [Show-o: One Single Transformer to Unify Multimodal Understanding and Generation](https://mlsanity.com/paper/2408.12528/) (2024)
- [Flow-GRPO: Training Flow Matching Models via Online RL](https://mlsanity.com/paper/2505.05470/) (2025)
- [Vector-quantized Image Modeling with Improved VQGAN](https://mlsanity.com/paper/2110.04627/) (2021)
- [Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning](https://mlsanity.com/paper/2506.01939/) (2025)
- [GIT: A Generative Image-to-text Transformer for Vision and Language](https://mlsanity.com/paper/2205.14100/) (2022)
- [Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model](https://mlsanity.com/paper/2408.11039/) (2024)
- [DanceGRPO: Unleashing GRPO on Visual Generation](https://mlsanity.com/paper/2505.07818/) (2025)
- [VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks](https://mlsanity.com/paper/2504.05118/) (2025)

## Built on by (most cited first, 17 of 17)

- [Flow-OPD: On-Policy Distillation for Flow Matching Models](https://mlsanity.com/paper/2605.08063/) (2026)
- [NextFlow: Unified Sequential Modeling Activates Multimodal Understanding and Generation](https://mlsanity.com/paper/2601.02204/) (2026)
- [OmniNFT: Modality-wise Omni Diffusion Reinforcement for Joint Audio-Video Generation](https://mlsanity.com/paper/2605.12480/) (2026)
- [ConsistentRFT: Reducing Visual Hallucinations in Flow-based Reinforcement Fine-Tuning](https://mlsanity.com/paper/2602.03425/) (2026)
- [Credit Where It is Due: Cross-Modality Connectivity Drives Precise Reinforcement Learning for MLLM Reasoning](https://mlsanity.com/paper/2602.11455/) (2026)
- [Power Reinforcement Post-Training of Text-to-Image Models with Super-Linear Advantage Shaping](https://mlsanity.com/paper/2605.10937/) (2026)
- [VAR RL Done Right: Tackling Asynchronous Policy Conflicts in Visual Autoregressive Generation](https://mlsanity.com/paper/2601.02256/) (2026)
- [Advances in GRPO for Generation Models: A Survey](https://mlsanity.com/paper/2603.06623/) (2026)
- [MAR-GRPO: Stabilized GRPO for AR-diffusion Hybrid Image Generation](https://mlsanity.com/paper/2604.06966/) (2026)
- [ParaUni: Enhance Generation in Unified Multimodal Model with Reinforcement-driven Hierarchical Parallel Information Interaction](https://mlsanity.com/paper/2512.05422/) (2025)
- [SJD-PAC: Accelerating Speculative Jacobi Decoding via Proactive Drafting and Adaptive Continuation](https://mlsanity.com/paper/2603.18599/) (2026)
- [RankE: End-to-End Post-Training for Discrete Text-to-Image Generation with Decoder Co-Evolution](https://mlsanity.com/paper/2605.21195/) (2026)
- [Policy-based Tuning of Autoregressive Image Models with Instance- and Distribution-Level Rewards](https://mlsanity.com/paper/2603.23086/) (2026)
- [Balancing Performance and Diversity in GRPO Autoregressive Text-to-Image Post-Training](https://mlsanity.com/paper/2606.21498/) (2026)
- [TTRSD: Test-Time Reinforcement Learning with Self-Distillation for Vision-Language Models](https://mlsanity.com/paper/2609.33414/) (2026)
- [Beyond Layers: Position-Resolved Gradient Conflict and Position-Aware Modulation for Unified Multimodal Models](https://mlsanity.com/paper/2609.38485/) (2026)
- [MaskSense: Confronting the Visual Exploration Trap in Masked Image Generation](https://mlsanity.com/paper/2610.17009/) (2026)

## Abstract (arXiv)

Recent studies have extended Reinforcement Learning with Verifiable Rewards (RLVR) to autoregressive (AR) visual generation and achieved promising progress. However, existing methods typically apply uniform optimization across all image tokens, while the varying contributions of different image tokens for RLVR's training remain unexplored. In fact, the key obstacle lies in how to identify more critical image tokens during AR generation and implement effective token-wise optimization for them. To tackle this challenge, we propose Group Critical-token Policy Optimization (GCPO), which facilitates effective policy optimization on critical tokens. We identify the critical tokens in RLVR-based AR generation from three perspectives, specifically: (1) Causal dependency: early tokens fundamentally determine the later tokens and final image effect due to unidirectional dependency; (2) Entropy-induced spatial structure: tokens with high entropy gradients correspond to image structure and bridges distinct visual regions; (3) RLVR-focused token diversity: tokens with low visual similarity across a group of sampled images contribute to richer token-level diversity. For these identified critical tokens, we further introduce a dynamic token-wise advantage weight to encourage exploration, based on confidence divergence between the policy model and reference model. By leveraging 30% of the image tokens, GCPO achieves better performance than GRPO with full tokens. Extensive experiments on multiple text-to-image benchmarks for both AR models and unified multimodal models demonstrate the effectiveness of GCPO for AR visual generation.