# GROW: Aligning GRPO with State-Action Modeling for Open-World VLM Agents

arXiv: https://arxiv.org/abs/2605.20246 · HTML: https://mlsanity.com/paper/2605.20246/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning.
- Builds on 21 of its 29 references that are in the map.
- First posted on arXiv 18 May 2026.

## Placed in

- LLM reasoning / [Reinforcement learning for reasoning](https://mlsanity.com/lr/reinforcement-learning-for-reasoning/)
- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)
- LLM agents and tool use / [Reinforcement learning for LLM agents](https://mlsanity.com/agents/reinforcement-learning-for-llm-agents/)

## Builds on (most cited first, 21 of 21)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models](https://mlsanity.com/paper/2403.13372/) (2024)
- [Interactive Language: Talking to Robots in Real Time](https://mlsanity.com/paper/2210.06407/) (2022)
- [AgentRL: Scaling Agentic Reinforcement Learning with a Multi-Turn, Multi-Task Framework](https://mlsanity.com/paper/2510.04206/) (2025)
- [MobileGUI-RL: Advancing Mobile GUI Agent through Reinforcement Learning in Online Environment](https://mlsanity.com/paper/2507.05720/) (2025)
- [MCU: An Evaluation Framework for Open-Ended Game Agents](https://mlsanity.com/paper/2310.08367/) (2023)
- [NitroGen: An Open Foundation Model for Generalist Gaming Agents](https://mlsanity.com/paper/2601.02427/) (2026)
- [Game-TARS: Pretrained Foundation Models for Scalable Generalist Multimodal Game Agents](https://mlsanity.com/paper/2510.23691/) (2025)
- [Lumine: An Open Recipe for Building Generalist Agents in 3D Open Worlds](https://mlsanity.com/paper/2511.08892/) (2025)
- [ActiveVLN: Towards Active Exploration via Multi-Turn RL in Vision-and-Language Navigation](https://mlsanity.com/paper/2509.12618/) (2025)
- [NaviMaster: Learning a Unified Policy for GUI and Embodied Navigation Tasks](https://mlsanity.com/paper/2508.02046/) (2025)
- [OpenHA: A Series of Open-Source Hierarchical Agentic Models in Minecraft](https://mlsanity.com/paper/2509.13347/) (2025)
- [ColorAgent: Building A Robust, Personalized, and Interactive OS Agent](https://mlsanity.com/paper/2510.19386/) (2025)
- [InquireMobile: Teaching VLM-based Mobile Agent to Request Human Assistance via Reinforcement Fine-Tuning](https://mlsanity.com/paper/2508.19679/) (2025)
- [ETP-R1: Evolving Topological Planning with Reinforcement Fine-tuning for Vision-Language Navigation in Continuous Environments](https://mlsanity.com/paper/2512.20940/) (2025)
- [LongNav-R1: Horizon-Adaptive Multi-Turn RL for Long-Horizon VLA Navigation](https://mlsanity.com/paper/2602.12351/) (2026)
- [Scalable Multi-Task Reinforcement Learning for Generalizable Spatial Intelligence in Visuomotor Agents](https://mlsanity.com/paper/2507.23698/) (2025)
- [MAIN-VLA: Modeling Abstraction of Intention and eNvironment for Vision-Language-Action Models](https://mlsanity.com/paper/2602.02212/) (2026)

## Abstract (arXiv)

Recently, vision-language model (VLM) agents have shown promising progress in open-world tasks, where successful task completion often requires multiple turns of visual perception and action execution. However, existing methods still rely primarily on Supervised Fine-Tuning (SFT) with expert demonstrations, while the advanced reinforcement learning (RL) algorithm, specifically Group Relative Policy Optimization (GRPO), has not been effectively employed for multi-turn RL in these tasks because standard GRPO requires full trajectories as training samples which leads to excessively long context and noise. To address this issue, we propose GROW, a RL framework for open-world VLM agents that decomposes collected trajectories into state-action samples, and computes advantages between these samples rather than treating a full trajectory as a single entity. We further provide a surrogate analysis indicating that, even though the grouped samples are conditioned on different local states rather than an identical prompt context, the objective can preserve the core relative policy optimization signal of GRPO under simplifying assumptions. Experiments on more than 800 Minecraft tasks show that our method achieves state-of-the-art (SOTA) performance, demonstrating the effectiveness of our proposed RL framework for open-world VLM agents.