# Scalable Multi-Task Reinforcement Learning for Generalizable Spatial Intelligence in Visuomotor Agents

arXiv: https://arxiv.org/abs/2507.23698 · HTML: https://mlsanity.com/paper/2507.23698/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use.
- Builds on 18 of its 30 references that are in the map.
- First posted on arXiv 31 Jul 2025.

## Placed in

- LLM agents and tool use / [Reinforcement learning for LLM agents](https://mlsanity.com/agents/reinforcement-learning-for-llm-agents/)
- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 18 of 18)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [Open X-Embodiment: Robotic Learning Datasets and RT-X Models](https://mlsanity.com/paper/2310.08864/) (2023)
- [Ray: A Distributed Framework for Emerging AI Applications](https://mlsanity.com/paper/1712.05889/) (2017)
- [MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge](https://mlsanity.com/paper/2206.08853/) (2022)
- [MineRL: A Large-Scale Dataset of Minecraft Demonstrations](https://mlsanity.com/paper/1907.13440/) (2019)
- [Mastering Complex Control in MOBA Games with Deep Reinforcement Learning](https://mlsanity.com/paper/1912.09729/) (2019)
- [STEVE-1: A Generative Model for Text-to-Behavior in Minecraft](https://mlsanity.com/paper/2306.00937/) (2023)
- [Navigating to Objects Specified by Images](https://mlsanity.com/paper/2304.01192/) (2023)
- [Visual Reinforcement Learning with Self-Supervised 3D Representations](https://mlsanity.com/paper/2210.07241/) (2022)
- [GROOT: Learning to Follow Instructions by Watching Gameplay Videos](https://mlsanity.com/paper/2310.08235/) (2023)
- [Open-World Multi-Task Control Through Goal-Aware Representation Learning and Adaptive Horizon Prediction](https://mlsanity.com/paper/2301.10034/) (2023)
- [RL-GPT: Integrating Reinforcement Learning and Code-as-policy](https://mlsanity.com/paper/2402.19299/) (2024)
- [UnrealZoo: Enriching Photo-realistic Virtual Worlds for Embodied AI](https://mlsanity.com/paper/2412.20977/) (2024)
- [Open-World Reinforcement Learning over Long Short-Term Imagination](https://mlsanity.com/paper/2410.03618/) (2024)
- [ROCKET-2: Steering Visuomotor Policy via Cross-View Goal Alignment](https://mlsanity.com/paper/2503.02505/) (2025)

## Built on by (most cited first, 3 of 3)

- [Self-supervised Hierarchical Visual Reasoning with World Model](https://mlsanity.com/paper/2605.17537/) (2026)
- [World2Minecraft: Occupancy-Driven Simulated Scenes Construction](https://mlsanity.com/paper/2604.27578/) (2026)
- [GROW: Aligning GRPO with State-Action Modeling for Open-World VLM Agents](https://mlsanity.com/paper/2605.20246/) (2026)

## Abstract (arXiv)

While Reinforcement Learning (RL) has achieved remarkable success in language modeling, its triumph hasn't yet fully translated to visuomotor agents. A primary challenge in RL models is their tendency to overfit specific tasks or environments, thereby hindering the acquisition of generalizable behaviors across diverse settings. This paper provides a preliminary answer to this challenge by demonstrating that RL-finetuned visuomotor agents in Minecraft can achieve zero-shot generalization to unseen worlds. Specifically, we explore RL's potential to enhance generalizable spatial reasoning and interaction capabilities in 3D worlds. To address challenges in multi-task RL representation, we analyze and establish cross-view goal specification as a unified multi-task goal space for visuomotor policies. Furthermore, to overcome the significant bottleneck of manual task design, we propose automated task synthesis within the highly customizable Minecraft environment for large-scale multi-task RL training, and we construct an efficient distributed RL framework to support this. Experimental results show RL significantly boosts interaction success rates by 4× and enables zero-shot generalization of spatial reasoning across diverse environments, including real-world settings. Our findings underscore the immense potential of RL training in 3D simulated environments, especially those amenable to large-scale task generation, for significantly advancing visuomotor agents' spatial reasoning.