# Robot-R1: Reinforcement Learning for Enhanced Embodied Reasoning in Robotics arXiv: https://arxiv.org/abs/2506.00070 · HTML: https://mlsanity.com/paper/2506.00070/ ## Key facts - Cited by 21 papers in the map (data through 14 Oct 2026). - Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. - Builds on 44 of its 70 references that are in the map. - First posted on arXiv 29 May 2025. ## Placed in - LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/) - Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/) ## Builds on (most cited first, 30 of 44) - [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024) - [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025) - [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022) - [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025) - [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024) - [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024) - [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023) - [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024) - [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025) - [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024) - [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022) - [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023) - [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022) - [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025) - [s1: Simple test-time scaling](https://mlsanity.com/paper/2501.19393/) (2025) - [Kimi k1.5: Scaling Reinforcement Learning with LLMs](https://mlsanity.com/paper/2501.12599/) (2025) - [Inner Monologue: Embodied Reasoning through Planning with Language Models](https://mlsanity.com/paper/2207.05608/) (2022) - [Gemini: A Family of Highly Capable Multimodal Models](https://mlsanity.com/paper/2312.11805/) (2023) - [VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models](https://mlsanity.com/paper/2307.05973/) (2023) - [Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models](https://mlsanity.com/paper/2503.06749/) (2025) - [VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model](https://mlsanity.com/paper/2504.07615/) (2025) - [SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild](https://mlsanity.com/paper/2503.18892/) (2025) - [Learning Interactive Real-World Simulators](https://mlsanity.com/paper/2310.06114/) (2023) - [Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models](https://mlsanity.com/paper/2405.01535/) (2024) - [CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation](https://mlsanity.com/paper/2411.19650/) (2024) - [Vision-Language Foundation Models as Effective Robot Imitators](https://mlsanity.com/paper/2311.01378/) (2023) - [Robotic Control via Embodied Chain-of-Thought Reasoning](https://mlsanity.com/paper/2407.08693/) (2024) - [3D-VLA: A 3D Vision-Language-Action Generative World Model](https://mlsanity.com/paper/2403.09631/) (2024) - [Latent Action Pretraining from Videos](https://mlsanity.com/paper/2410.11758/) (2024) - [SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models](https://mlsanity.com/paper/2406.01584/) (2024) ## Built on by (most cited first, 21 of 21) - [A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agents](https://mlsanity.com/paper/2601.12538/) (2026) - [Fast-ThinkAct: Efficient Vision-Language-Action Reasoning via Verbalizable Latent Planning](https://mlsanity.com/paper/2601.09708/) (2026) - [DeepThinkVLA: Enhancing Reasoning Capability of Vision-Language-Action Models](https://mlsanity.com/paper/2511.15669/) (2025) - [Parallels Between VLA Model Post-Training and Human Motor Learning: Progress, Challenges, and Trends](https://mlsanity.com/paper/2506.20966/) (2025) - [On-the-Fly VLA Adaptation via Test-Time Reinforcement Learning](https://mlsanity.com/paper/2601.06748/) (2026) - [Towards Understanding Visual Grounding in Visual Language Models](https://mlsanity.com/paper/2509.10345/) (2025) - [Why Tree-Style Branching Matters for Thought Advantage Estimation in GRPO](https://mlsanity.com/paper/2509.24494/) (2025) - [ERA: Transforming VLMs into Embodied Agents via Embodied Prior Learning and Online Reinforcement Learning](https://mlsanity.com/paper/2510.12693/) (2025) - [Reinforcement Learning for Large Model: A Survey](https://mlsanity.com/paper/2508.08189/) (2025) - [PanoEnv: Exploring 3D Spatial Intelligence in Panoramic Environments with Reinforcement Learning](https://mlsanity.com/paper/2602.21992/) (2026) - [Revisiting Embodied Chain-of-Thought for Generalizable Robot Manipulation](https://mlsanity.com/paper/2606.03784/) (2026) - [Toward Ambulatory Vision: Learning Visually-Grounded Active View Selection](https://mlsanity.com/paper/2512.13250/) (2025) - [VLA-Thinker: Boosting Vision-Language-Action Models through Thinking-with-Image Reasoning](https://mlsanity.com/paper/2603.14523/) (2026) - [Notes-to-Self: Scratchpad Augmented VLAs for Memory Dependent Manipulation Tasks](https://mlsanity.com/paper/2602.21013/) (2026) - [RoboAlign: Learning Test-Time Reasoning for Language-Action Alignment in Vision-Language-Action Models](https://mlsanity.com/paper/2603.21341/) (2026) - [ReRec: Reasoning-Augmented LLM-based Recommendation Assistant via Reinforcement Fine-tuning](https://mlsanity.com/paper/2604.07851/) (2026) - [AsyncWebRL: Efficient Asynchronous Reinforcement Learning for Multi-Step Visual Web Agents](https://mlsanity.com/paper/2606.05597/) (2026) - [Towards Precise Intent-Aligned VLA Aerial Navigation via Expert-Guided GRPO](https://mlsanity.com/paper/2606.02313/) (2026) - [An Efficient Training Pipeline for Reasoning Graphical User Interface Agents](https://mlsanity.com/paper/2511.08172/) (2025) - [Collaborative Weighting with Pessimistic Critic for Mitigating Overestimation in Off-Policy Reinforcement Learning](https://mlsanity.com/paper/2607.26509/) (2026) - [TrustRoboReward: Preference-Ordered Isotonic Score Editing for Multi-Paradigm Robot Reward Models](https://mlsanity.com/paper/2608.08491/) (2026) ## Abstract (arXiv) Large Vision-Language Models (LVLMs) have recently shown great promise in advancing robotics by combining embodied reasoning with robot control. A common approach involves training on embodied reasoning tasks related to robot control using Supervised Fine-Tuning (SFT). However, SFT datasets are often heuristically constructed and not explicitly optimized for improving robot control. Furthermore, SFT often leads to issues such as catastrophic forgetting and reduced generalization performance. To address these limitations, we introduce Robot-R1, a novel framework that leverages reinforcement learning to enhance embodied reasoning specifically for robot control. Robot-R1 learns to predict the next keypoint state required for task completion, conditioned on the current scene image and environment metadata derived from expert demonstrations. Inspired by the DeepSeek-R1 learning approach, Robot-R1 samples reasoning-based responses and reinforces those that lead to more accurate predictions. To rigorously evaluate Robot-R1, we also introduce a new benchmark that demands the diverse embodied reasoning capabilities for the task. Our experiments show that models trained with Robot-R1 outperform SFT methods on embodied reasoning tasks. Despite having only 7B parameters, Robot-R1 even surpasses GPT-4o on reasoning tasks related to low-level action control, such as spatial and movement reasoning.