mlsanityData to 13 Oct 2026

RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Robotics

arXiv 2406.10721 ↗First posted 15 Jun 2024cs.RO cs.AI cs.CVcited by 243

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

From rearranging objects on a table to putting groceries into shelves, robots must plan precise action points to perform tasks accurately and reliably. In spite of the recent adoption of vision language models (VLMs) to control robot behavior, VLMs struggle to precisely articulate robot actions using language. We introduce an automatic synthetic data generation pipeline that instruction-tunes VLMs to robotic domains and needs. Using the pipeline, we train RoboPoint, a VLM that predicts image keypoint affordances given language instructions. Compared to alternative approaches, our method requires no real-world data collection or human demonstration, making it much more scalable to diverse environments and viewpoints. In addition, RoboPoint is a general model that enables several downstream applications such as robot navigation, manipulation, and augmented reality (AR) assistance. Our experiments demonstrate that RoboPoint outperforms state-of-the-art VLMs (GPT-4o) and visual prompting techniques (PIVOT) by 21.8% in the accuracy of predicting spatial affordance and by 30.5% in the success rate of downstream tasks. Project website: https://robo-point.github.io.

Key facts. Cited by 243 papers in the map (data through 13 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 49 of its 61 references that are in the map. First posted on arXiv 15 Jun 2024.

Builds on

49 of 61 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs12 Feb
  2. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities22 Jan
  3. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul

Other references · 46

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  2. 2020Language Models are Few-Shot Learnerscited by 32,320
  3. 2023GPT-4 Technical Reportcited by 18,278
  4. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,649
  5. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,506
  6. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,321
  7. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,028
  8. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,709
  9. 2019Towards VQA Models That Can Readcited by 1,705
  10. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,637
  11. 2024Octo: An Open-Source Generalist Robot Policycited by 1,450
  12. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,424
34 more
  1. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  2. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,186
  3. 2019Habitat: A Platform for Embodied AI Researchcited by 1,169
  4. 2016Modeling Context in Referring Expressionscited by 1,057
  5. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 933
  6. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  7. 2017PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenescited by 770
  8. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 739
  9. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 602
  10. 2020SAPIEN: A SimulAted Part-based Interactive ENvironmentcited by 588
  11. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 546
  12. 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 478
  13. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 371
  14. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 342
  15. 2023RVT: Robotic View Transformer for 3D Object Manipulationcited by 225
  16. 2021Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenescited by 216
  17. 2021Where2Act: From Pixels to Actions for Articulated 3D Objectscited by 181
  18. 2019kPAM: KeyPoint Affordances for Category-Level Robotic Manipulationcited by 152
  19. 2018Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulationcited by 140
  20. 2017AffordanceNet: An End-to-End Deep Learning Approach for Object Affordance Detectioncited by 126
  21. 2024MOKA: Open-World Robotic Manipulation through Mark-Based Visual Promptingcited by 126
  22. 2020ACRONYM: A Large-Scale Grasp Dataset Based on Simulationcited by 116
  23. 2021Synergies Between Affordance and Geometry: 6-DoF Grasp Detection via Implicit Representationscited by 107
  24. 2021ManipulaTHOR: A Framework for Visual Object Manipulationcited by 102
  25. 2019KETO: Learning Keypoint Representations for Tool Manipulationcited by 68
  26. 2021StructFormer: Learning Spatial Structure for Language-Guided Semantic Rearrangement of Novel Objectscited by 57
  27. 2022Motion Policy Networkscited by 53
  28. 2021SORNet: Spatial Object-Centric Representations for Sequential Manipulationcited by 48
  29. 2023VL-Grasp: a 6-Dof Interactive Grasp Policy for Language-Oriented Objects in Cluttered Indoor Scenescited by 40
  30. 2023M2T2: Multi-Task Masked Transformer for Object-centric Pick and Placecited by 40
  31. 2020Same Object, Different Grasps: Data and Semantic Knowledge for Task-Oriented Graspingcited by 26
  32. 2023AR2-D2:Training a Robot Without a Robotcited by 26
  33. 2022A Survey on Machine Learning Approaches for Modelling Intuitive Physicscited by 20
  34. 2023CabiNet: Scaling Neural Collision Detection for Object Rearrangement with Procedural Scene Generationcited by 17

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 19; 2025: 101; 2026: 123.

243 papers.

Sort:

By area:

  1. 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 308
  2. 2024TraceVLA: Visual Trace Prompting Enhances Spatial-Temporal Awareness for Generalist Robotic Policiescited by 302
  3. 2025DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledgecited by 207
  4. 2025ThinkAct: Vision-Language-Action Reasoning via Reinforced Visual Latent Planningcited by 153
  5. 2026Cosmos 3: Omnimodal World Models for Physical AIcited by 153
  6. 2024RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Roboticscited by 133
  7. 2025HAMSTER: Hierarchical Action Models For Open-World Robot Manipulationcited by 125
  8. 2024SAT: Dynamic Spatial Aptitude Training for Multimodal Language Modelscited by 111
  9. 2025OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Modelscited by 110
  10. 2025OneTwoVLA: A Unified Vision-Language-Action Model with Adaptive Reasoningcited by 106
  11. 2025Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videoscited by 97
  12. 2025InternVLA-M1: A Spatially Guided Vision-Language-Action Framework for Generalist Robot Policycited by 83
38 more
  1. 2024Manipulate-Anything: Automating Real-World Robots using Vision-Language Modelscited by 82
  2. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  3. 2025BridgeVLA: Input-Output Alignment for Efficient 3D Manipulation Learning with Vision-Language Modelscited by 79
  4. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 68
  5. 2025Detect Anything via Next Point Predictioncited by 66
  6. 2026Being-H0.5: Scaling Human-Centric Robot Learning for Cross-Embodiment Generalizationcited by 64
  7. 2025SpatialCoT: Advancing Spatial Reasoning through Coordinate Alignment and Chain-of-Thought for Embodied Task Planningcited by 62
  8. 2024Code-as-Monitor: Constraint-aware Visual Programming for Reactive and Proactive Robotic Failure Detectioncited by 61
  9. 2024Do Vision-Language Models Represent Space and How? Evaluating Spatial Frame of Reference Under Ambiguitiescited by 58
  10. 2024Affordance-based Robot Manipulation with Flow Matchingcited by 54
  11. 2024LLaRA: Supercharging Robot Learning Data for Vision-Language Policycited by 51
  12. 2025FlowVLA: Visual Chain of Thought-based Motion Reasoning for Vision-Language-Action Modelscited by 50
  13. 2026Qwen-RobotManip Technical Report: Alignment Unlocks Scale for Robotic Manipulation Foundation Modelscited by 50
  14. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 47
  15. 2026MolmoAct2: Action Reasoning Models for Real-world Deploymentcited by 47
  16. 2025SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulationcited by 46
  17. 2025SSR: Enhancing Depth Perception in Vision-Language Models via Rationale-Guided Spatial Reasoningcited by 45
  18. 2025How Far are VLMs from Visual Spatial Intelligence? A Benchmark-Driven Perspectivecited by 42
  19. 2024RT-Affordance: Affordances are Versatile Intermediate Representations for Robot Manipulationcited by 41
  20. 2026VLM4VLA: Revisiting Vision-Language-Models in Vision-Language-Action Modelscited by 39
  21. 2025Embodied-R1: Reinforced Embodied Reasoning for General Robotic Manipulationcited by 37
  22. 2025From Seeing to Doing: Bridging Reasoning and Decision for Robotic Manipulationcited by 34
  23. 2025VLA-OS: Structuring and Dissecting Planning Representations and Paradigms in Vision-Language-Action Modelscited by 34
  24. 2025FailSafe: Reasoning and Recovery from Failures in Vision-Language-Action Modelscited by 34
  25. 20253D Aware Region Prompted Vision Language Modelcited by 32
  26. 2025Multimodal Spatial Reasoning in the Large Model Era: A Survey and Benchmarkscited by 32
  27. 2025PointArena: Probing Multimodal Grounding Through Language-Guided Pointingcited by 31
  28. 2024CoA-VLA: Improving Vision-Language-Action Models via Visual-Textual Chain-of-Affordancecited by 30
  29. 2025RoboOS: A Hierarchical Embodied Framework for Cross-Embodiment and Multi-Agent Collaborationcited by 28
  30. 2025Spatial Understanding from Videos: Structured Prompts Meet Simulation Datacited by 28
  31. 2025TIGeR: Tool-Integrated Geometric Reasoning in Vision-Language Models for Roboticscited by 27
  32. 2024Steering Large Language Models between Code Execution and Textual Reasoningcited by 24
  33. 2025ManipBench: Benchmarking Vision-Language Models for Low-Level Robot Manipulationcited by 24
  34. 2025Enhancing Generalization in Vision-Language-Action Models by Preserving Pretrained Representationscited by 24
  35. 2025Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Braincited by 24
  36. 2025UAD: Unsupervised Affordance Distillation for Generalization in Robotic Manipulationcited by 23
  37. 2025Robix: A Unified Model for Robot Interaction, Reasoning and Planningcited by 23
  38. 2025Vlaser: Vision-Language-Action Model with Synergistic Embodied Reasoningcited by 23

50 most cited of 243 All 243 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Spa-Bench: A Real-Robot Benchmark for Spatial Grounding in Vision-Language-Action Models
  2. 2025Affordance Field Intervention: Enabling VLAs to Escape Memory Traps in Robotic Manipulation
  3. 2025RoboAfford++: A Generative AI-Enhanced Dataset for Multimodal Affordance Learning in Robotic Manipulation and Navigation
  4. 2026AffordanceVLA: A Vision-Language-Action Model Empowering Action Generation through Affordance-Aware Understanding
  5. 2024KALIE: Fine-Tuning Vision-Language Models for Open-World Manipulation without Robot Data
  6. 2026RoboIRGBench: Benchmarking Implicit Referential Grounding in Vision-Language-Action Models
  7. 2024Affordance-Guided Reinforcement Learning via Visual Prompting
  8. 2025Seeing Across Views: Benchmarking Spatial Reasoning of Vision-Language Models in Robotic Scenes