mlsanityData to 7 Oct 2026

RoboRefer: Towards Spatial Referring with Reasoning in Vision-Language Models for Robotics

arXiv 2506.04308 ↗First posted 4 Jun 2025cs.RO cs.AI cs.CVcited by 125

On the map: LLM reasoning → Spatial and embodied reasoning

also in 1 more
Abstract (arXiv)

Spatial referring is a fundamental capability of embodied robots to interact with the 3D physical world. However, even with the powerful pretrained vision language models (VLMs), recent approaches are still not qualified to accurately understand the complex 3D scenes and dynamically reason about the instruction-indicated locations for interaction. To this end, we propose RoboRefer, a 3D-aware VLM that can first achieve precise spatial understanding by integrating a disentangled but dedicated depth encoder via supervised fine-tuning (SFT). Moreover, RoboRefer advances generalized multi-step spatial reasoning via reinforcement fine-tuning (RFT), with metric-sensitive process reward functions tailored for spatial referring tasks. To support SFT and RFT training, we introduce RefSpatial, a large-scale dataset of 20M QA pairs (2x prior), covering 31 spatial relations (vs. 15 prior) and supporting complex reasoning processes (up to 5 steps). In addition, we introduce RefSpatial-Bench, a challenging benchmark filling the gap in evaluating spatial referring with multi-step reasoning. Experiments show that SFT-trained RoboRefer achieves state-of-the-art spatial understanding, with an average success rate of 89.6%. RFT-trained RoboRefer further outperforms all other baselines by a large margin, even surpassing Gemini-2.5-Pro by 17.4% in average accuracy on RefSpatial-Bench. Notably, RoboRefer can be integrated with various control policies to execute long-horizon, dynamic tasks across diverse robots (e,g., UR5, G1 humanoid) in cluttered real-world scenes.

Key facts. Cited by 125 papers in the map (data through 7 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 20 of its 159 references that are in the map. First posted on arXiv 4 Jun 2025.

Builds on

20 of 159 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025ManipLVM-R1: Reinforcement Learning for Reasoning in Embodied Manipulation with Large Vision-Language Models22 May
  2. 2025SSR: Enhancing Depth Perception in Vision-Language Models via Rationale-Guided Spatial Reasoning18 May

Other references · 18

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 867
  3. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Modelcited by 515
  4. 2025RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concretecited by 134
  5. 2024Code-as-Monitor: Constraint-aware Visual Programming for Reactive and Proactive Robotic Failure Detectioncited by 59
  6. 2024LaMP: Language-Motion Pretraining for Motion Generation, Retrieval, and Captioningcited by 58
  7. 2024MineDreamer: Learning to Follow Instructions via Chain-of-Imagination for Simulated-World Controlcited by 52
  8. 2025T2ISafety: Benchmark for Assessing Fairness, Toxicity, and Privacy in Image Generationcited by 47
  9. 2025TrackVLA: Embodied Visual Tracking in the Wildcited by 46
  10. 2024Cubify Anything: Scaling Indoor 3D Object Detectioncited by 39
  11. 2025RoboFactory: Exploring Embodied Agent Collaboration with Compositional Constraintscited by 31
  12. 2025ReSo: A Reward-driven Self-organizing LLM-based Multi-Agent System for Reasoning Taskscited by 30
6 more
  1. 2025RoboOS: A Hierarchical Embodied Framework for Cross-Embodiment and Multi-Agent Collaborationcited by 28
  2. 2024RH20T-P: A Primitive-Level Robotic Dataset Towards Composable Generalization Agentscited by 20
  3. 2025LabUtopia: High-Fidelity Simulation and Hierarchical Benchmark for Scientific Embodied Agentscited by 14
  4. 2025NavigateDiff: Visual Predictors are Zero-Shot Navigation Assistantscited by 11
  5. 2024Story3D-Agent: Exploring 3D Storytelling Visualization with Large Language Modelscited by 9
  6. 2024MulSMo: Multimodal Stylized Motion Generation by Bidirectional Control Flowcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 41; 2026: 84.

125 papers.

Sort:

By area:

  1. 2025Qwen3-VL Technical Reportcited by 2,534
  2. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  3. 2025InternVLA-M1: A Spatially Guided Vision-Language-Action Framework for Generalist Robot Policycited by 83
  4. 2025Detect Anything via Next Point Predictioncited by 66
  5. 2026Being-H0.5: Scaling Human-Centric Robot Learning for Cross-Embodiment Generalizationcited by 61
  6. 2026Seed1.8 Model Card: Towards Generalized Real-World Agencycited by 47
  7. 2026Qwen-RobotManip Technical Report: Alignment Unlocks Scale for Robotic Manipulation Foundation Modelscited by 47
  8. 2026MolmoAct2: Action Reasoning Models for Real-world Deploymentcited by 45
  9. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 41
  10. 2026VLM4VLA: Revisiting Vision-Language-Models in Vision-Language-Action Modelscited by 38
  11. 2025Robo-Dopamine: General Process Reward Modeling for High-Precision Robotic Manipulationcited by 37
  12. 2025Embodied-R1: Reinforced Embodied Reasoning for General Robotic Manipulationcited by 36
38 more
  1. 2026LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligencecited by 31
  2. 2025Holistic Evaluation of Multimodal LLMs on Spatial Intelligencecited by 28
  3. 2025TIGeR: Tool-Integrated Geometric Reasoning in Vision-Language Models for Roboticscited by 27
  4. 2025Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Braincited by 24
  5. 2025Vlaser: Vision-Language-Action Model with Synergistic Embodied Reasoningcited by 22
  6. 2025Spatial Reasoning in Multimodal Large Language Models: A Survey of Tasks, Benchmarks and Methodscited by 22
  7. 2026Think3D: Thinking with Space for Spatial Reasoningcited by 22
  8. 2025SpaceTools: Tool-Augmented Spatial Reasoning via Double Interactive RLcited by 20
  9. 2026A Systematic Study of Data Modalities and Strategies for Co-training Large Behavior Models for Robot Manipulationcited by 20
  10. 2025Seeing Across Views: Benchmarking Spatial Reasoning of Vision-Language Models in Robotic Scenescited by 19
  11. 2025VIKI-R: Coordinating Embodied Multi-Agent Cooperation via Reinforcement Learningcited by 18
  12. 2025Reasoning in Space via Grounding in the Worldcited by 18
  13. 2026Theory of Space: Can Foundation Models Construct Spatial Beliefs through Active Exploration?cited by 18
  14. 2026HY-Embodied-0.5: Embodied Foundation Models for Real-World Agentscited by 18
  15. 2025From Language to Locomotion: Retargeting-free Humanoid Control via Motion Latent Guidancecited by 17
  16. 2025DrivePI: Spatial-aware 4D MLLM for Unified Autonomous Driving Understanding, Perception, Prediction and Planningcited by 15
  17. 2026TwinBrainVLA: Unleashing the Potential of Generalist VLMs for Embodied Tasks via Asymmetric Mixture-of-Transformerscited by 14
  18. 2026Thinking with Geometry: Active Geometry Integration for Spatial Reasoningcited by 14
  19. 2025Towards Cross-View Point Correspondence in Vision-Language Modelscited by 12
  20. 2025Towards Spatial Trace with Reasoning in Vision-Language Models for Roboticscited by 12
  21. 2026RoboInter: A Holistic Intermediate Representation Suite Towards Robotic Manipulationcited by 12
  22. 2025Prune4Web: DOM Tree Pruning Programming for Web Agentcited by 11
  23. 2026ReVSI: Rebuilding Visual Spatial Intelligence Evaluation for Accurate Assessment of VLM 3D Reasoningcited by 11
  24. 2025NaviMaster: Learning a Unified Policy for GUI and Embodied Navigation Taskscited by 10
  25. 2025Point-It-Out: Benchmarking Embodied Reasoning for Vision Language Models in Multi-Stage Visual Groundingcited by 10
  26. 2025Is your VLM Sky-Ready? A Comprehensive Spatial Intelligence Benchmark for UAV Navigationcited by 10
  27. 2025Point What You Mean: Visually Grounded Instruction Policycited by 10
  28. 2025VisualTrans: A Benchmark for Real-World Visual Transformation Reasoningcited by 9
  29. 2025COOPER: A Unified Model for Cooperative Perception and Reasoning in Spatial Intelligencecited by 9
  30. 2025Dissecting Embodied Abilities in Multimodal Language Models through Skill-level Evaluation and Diagnosiscited by 8
  31. 2026ST4VLA: Spatially Guided Training for Vision-Language-Action Modelscited by 8
  32. 2026Qwen-Drive-1.0: An Initial Step towards a Vision-Language Foundation Model for Autonomous Drivingcited by 8
  33. 2025Mind to Hand: Purposeful Robotic Control via Embodied Reasoningcited by 7
  34. 2026Learning Actionable Manipulation Recovery via Counterfactual Failure Synthesiscited by 7
  35. 2026Hy-Embodied-VLM-1.0: Efficient Physical-World Agentscited by 7
  36. 2025RoboMirror: Understand Before You Imitate for Video to Humanoid Locomotioncited by 6
  37. 2026Green-VLA: Staged Vision-Language-Action Model for Generalist Robotscited by 6
  38. 2026SaPaVe: Towards Active Perception and Manipulation in Vision-Language-Action Models for Roboticscited by 6

50 most cited of 125 All 125 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Robotics
  2. 2026Technical Report of RoboSpatial Challenge at CVPR 2026: Selective Reasoning Activation and Reference-Frame Disambiguation for Embodied Spatial Reasoning
  3. 2025Robot-R1: Reinforcement Learning for Enhanced Embodied Reasoning in Robotics
  4. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models
  5. 2025Spatial Understanding from Videos: Structured Prompts Meet Simulation Data
  6. 2026RoboIRGBench: Benchmarking Implicit Referential Grounding in Vision-Language-Action Models
  7. 2025RoboGPT-R1: Enhancing Robot Task Planning with Reinforcement Learning
  8. 2025From Seeing to Doing: Bridging Reasoning and Decision for Robotic Manipulation