mlsanityData to 7 Oct 2026

MOKA: Open-World Robotic Manipulation through Mark-Based Visual Prompting

arXiv 2403.03174 ↗First posted 5 Mar 2024cs.RO cs.AIcited by 126

Not on the map. Why?

Cited by 31 in Vision-language-action models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Open-world generalization requires robotic systems to have a profound understanding of the physical world and the user command to solve diverse and complex tasks. While the recent advancement in vision-language models (VLMs) has offered unprecedented opportunities to solve open-world problems, how to leverage their capabilities to control robots remains a grand challenge. In this paper, we introduce Marking Open-world Keypoint Affordances (MOKA), an approach that employs VLMs to solve robotic manipulation tasks specified by free-form language instructions. Central to our approach is a compact point-based representation of affordance, which bridges the VLM's predictions on observed images and the robot's actions in the physical world. By prompting the pre-trained VLM, our approach utilizes the VLM's commonsense knowledge and concept understanding acquired from broad data sources to predict affordances and generate motions. To facilitate the VLM's reasoning in zero-shot and few-shot manners, we propose a visual prompting technique that annotates marks on images, converting affordance reasoning into a series of visual question-answering problems that are solvable by the VLM. We further explore methods to enhance performance with robot experiences collected by MOKA through in-context learning and policy distillation. We evaluate and analyze MOKA's performance on various table-top manipulation tasks including tool use, deformable body manipulation, and object rearrangement.

Key facts. Cited by 126 papers in the map (data through 7 Oct 2026). Builds on 52 of its 67 references that are in the map. First posted on arXiv 5 Mar 2024.

Builds on

52 of 67 references.

References · 52

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2023Segment Anythingcited by 8,158
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  11. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  12. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
38 more
  1. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  2. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  3. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  4. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  5. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  6. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  7. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  8. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  9. 2021Grounded Language-Image Pre-trainingcited by 1,147
  10. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  11. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  12. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  13. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  14. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  15. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  16. 2023Segment Everything Everywhere All at Oncecited by 546
  17. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 542
  18. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  19. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  20. 2023DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editingcited by 293
  21. 2016Deep Visual Foresight for Planning Robot Motioncited by 276
  22. 2017Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metricscited by 259
  23. 2023Reward Design with Language Modelscited by 257
  24. 2023Text2Motion: From Natural Language Instructions to Feasible Planscited by 223
  25. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 221
  26. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 208
  27. 2022CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigationcited by 194
  28. 2023Semantic-SAM: Segment and Recognize Anything at Any Granularitycited by 185
  29. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planningcited by 161
  30. 2022Open-vocabulary Queryable Scene Representations for Real World Planningcited by 159
  31. 2023Open-World Object Manipulation using Pre-trained Vision-Language Modelscited by 155
  32. 2019kPAM: KeyPoint Affordances for Category-Level Robotic Manipulationcited by 152
  33. 2021Synergies Between Affordance and Geometry: 6-DoF Grasp Detection via Implicit Representationscited by 107
  34. 2019Scene Memory Transformer for Embodied Agents in Long-Horizon Taskscited by 106
  35. 2023Fine-Grained Visual Promptingcited by 81
  36. 2019KETO: Learning Keypoint Representations for Tool Manipulationcited by 68
  37. 2023Goal Representations for Instruction Following: A Semi-Supervised Language Interface to Controlcited by 32
  38. 2022Planning to Practice: Efficient Online Fine-Tuning by Composing Goals in Latent Spacecited by 31

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 21; 2025: 45; 2026: 60.

126 papers.

Sort:

By area:

  1. 2024RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Roboticscited by 242
  2. 2025DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledgecited by 203
  3. 2024RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Roboticscited by 133
  4. 2025OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Modelscited by 105
  5. 2024In-Context Imitation Learning via Next-Token Predictioncited by 65
  6. 2024RAM: Retrieval-Based Affordance Transfer for Generalizable Zero-Shot Robotic Manipulationcited by 55
  7. 2025RoboGround: Robotic Manipulation with Grounded Vision-Language Priorscited by 50
  8. 2025SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulationcited by 46
  9. 2024RT-Affordance: Affordances are Versatile Intermediate Representations for Robot Manipulationcited by 40
  10. 2025Embodied-R1: Reinforced Embodied Reasoning for General Robotic Manipulationcited by 36
  11. 2024A3VLM: Actionable Articulation-Aware Vision Language Modelcited by 33
  12. 2024LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrationscited by 32
38 more
  1. 2024BUMBLE: Unifying Reasoning and Acting with Vision-Language Models for Building-wide Mobile Manipulationcited by 31
  2. 2025ManipBench: Benchmarking Vision-Language Models for Low-Level Robot Manipulationcited by 24
  3. 2024Affordance-Guided Reinforcement Learning via Visual Promptingcited by 20
  4. 2026PALM: Progress-Aware Policy Learning via Affordance Reasoning for Long-Horizon Robotic Manipulationcited by 20
  5. 2024Keypoint Abstraction using Large Models for Object-Relative Imitation Learningcited by 18
  6. 2025See, Point, Fly: A Learning-Free VLM Framework for Universal Unmanned Aerial Navigationcited by 18
  7. 2026RHO: Your Coding Agent is Secretly a Roboticistcited by 18
  8. 2026LAP: Language-Action Pre-Training Enables Zero-shot Cross-Embodiment Transfercited by 17
  9. 2024KALIE: Fine-Tuning Vision-Language Models for Open-World Manipulation without Robot Datacited by 15
  10. 2026MaskWAM: Unifying Mask Prompting and Prediction for World-Action Modelscited by 15
  11. 2024Blox-Net: Generative Design-for-Robot-Assembly Using VLM Supervision, Physics Simulation, and a Robot with Resetcited by 14
  12. 2024Towards Open-World Grasping with Large Vision-Language Modelscited by 13
  13. 2024VL-TGS: Trajectory Generation and Selection using Vision Language Models in Mapless Outdoor Environmentscited by 11
  14. 2025Point What You Mean: Visually Grounded Instruction Policycited by 10
  15. 2025Clutter-Robust Vision-Language-Action Models through Object-Centric and Geometry Groundingcited by 10
  16. 2025EmbodiedVSR: Dynamic Scene Graph-Guided Chain-of-Thought Reasoning for Visual Spatial Taskscited by 9
  17. 2025RoboChemist: Long-Horizon and Safety-Compliant Robotic Chemical Experimentationcited by 9
  18. 2025Free-form language-based robotic reasoning and graspingcited by 8
  19. 2025Maestro: Orchestrating Robotics Modules with Vision-Language Models for Zero-Shot Generalist Robotscited by 8
  20. 2026ProcVLM: Learning Procedure-Grounded Progress Rewards for Robotic Manipulationcited by 8
  21. 2025Scaffolding Dexterous Manipulation with Vision-Language Modelscited by 7
  22. 2025SIMPACT: Simulation-Enabled Action Planning using Vision-Language Modelscited by 7
  23. 2026NovaPlan: Zero-Shot Long-Horizon Manipulation via Closed-Loop Video Language Planningcited by 7
  24. 2026Point Tracking Improves World Action Modelscited by 7
  25. 2024One-Shot Manipulation Strategy Learning by Making Contact Analogiescited by 6
  26. 2025A Data-Centric Revisit of Pre-Trained Vision Models for Robot Learningcited by 6
  27. 2025Spatial RoboGrasp: Generalized Robotic Grasping Control Policycited by 6
  28. 2025Seeing is Not Reasoning: MVPBench for Graph-based Evaluation of Multi-path Visual Physical CoTcited by 6
  29. 2025Learning Affordances at Inference-Time for Vision-Language-Action Modelscited by 6
  30. 2026PLanAR: Planning-Language-Grounded Agentic Reasoning for Robot Manipulationcited by 6
  31. 2026Libra-VLA: Achieving Learning Equilibrium via Asynchronous Coarse-to-Fine Dual-Systemcited by 6
  32. 2026ETA: A New Agentic Paradigm for Embodied Taskscited by 6
  33. 2024Robot Manipulation in Salient Vision through Referring Image Segmentation and Geometric Constraintscited by 5
  34. 2024Vision Language Models Can Parse Floor Plan Mapscited by 5
  35. 2024Scene Exploration by Vision-Language Modelscited by 5
  36. 2024Visual Prompting with Iterative Refinement for Design Critique Generationcited by 5
  37. 2025Visual Prompting for Robotic Manipulation with Annotation-Guided Pick-and-Place Using ACTcited by 5
  38. 2025Social 3D Scene Graphs: Modeling Human Actions and Relations for Interactive Service Robotscited by 5

50 most cited of 126 All 126 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024CoA-VLA: Improving Vision-Language-Action Models via Visual-Textual Chain-of-Affordance
  2. 2025AffordGrasp: In-Context Affordance Reasoning for Open-Vocabulary Task-Oriented Grasping in Clutter
  3. 2024ManipVQA: Injecting Robotic Affordance and Physically Grounded Information into Multi-Modal Large Language Models
  4. 2025From Seeing to Doing: Bridging Reasoning and Decision for Robotic Manipulation
  5. 2025OmniManip: Towards General Robotic Manipulation via Object-Centric Interaction Primitives as Spatial Constraints
  6. 2025TRACE: Textual Reasoning for Affordance Coordinate Extraction
  7. 2025Prompting with the Future: Open-World Model Predictive Control with Interactive Digital Twins
  8. 2024PAVLM: Advancing Point Cloud based Affordance Understanding Via Vision-Language Model