mlsanityData to 14 Oct 2026

ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation

arXiv 2409.01652 ↗First posted 3 Sep 2024cs.RO cs.AI cs.CVcited by 315

On the map: LLM reasoning → Spatial and embodied reasoning

Abstract (arXiv)

Representing robotic manipulation tasks as constraints that associate the robot and the environment is a promising way to encode desired robot behaviors. However, it remains unclear how to formulate the constraints such that they are 1) versatile to diverse tasks, 2) free of manual labeling, and 3) optimizable by off-the-shelf solvers to produce robot actions in real-time. In this work, we introduce Relational Keypoint Constraints (ReKep), a visually-grounded representation for constraints in robotic manipulation. Specifically, ReKep is expressed as Python functions mapping a set of 3D keypoints in the environment to a numerical cost. We demonstrate that by representing a manipulation task as a sequence of Relational Keypoint Constraints, we can employ a hierarchical optimization procedure to solve for robot actions (represented by a sequence of end-effector poses in SE(3)) with a perception-action loop at a real-time frequency. Furthermore, in order to circumvent the need for manual specification of ReKep for each new task, we devise an automated procedure that leverages large vision models and vision-language models to produce ReKep from free-form language instructions and RGB-D observations. We present system implementations on a wheeled single-arm platform and a stationary dual-arm platform that can perform a large variety of manipulation tasks, featuring multi-stage, in-the-wild, bimanual, and reactive behaviors, all without task-specific data or environment models. Website at https://rekep-robot.github.io/.

Key facts. Cited by 315 papers in the map (data through 14 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 112 of its 164 references that are in the map. First posted on arXiv 3 Sep 2024.

Builds on

112 of 164 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Grounding Language Plans in Demonstrations Through Counterfactual Perturbations25 Mar
  2. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs12 Feb
  3. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities22 Jan
  4. 2023ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulation24 Dec

Other references · 108

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2023Segment Anythingcited by 8,234
  6. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  10. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  11. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  12. 2020Object-Centric Learning with Slot Attentioncited by 876
38 more
  1. 2023Vision Transformers Need Registerscited by 871
  2. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 750
  3. 2023Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesiscited by 737
  4. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  5. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  6. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 529
  7. 2023CoTracker: It is Better to Track Togethercited by 522
  8. 2016Interaction Networks for Learning about Objects, Relations and Physicscited by 509
  9. 2023FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objectscited by 429
  10. 2019MONet: Unsupervised Scene Decomposition and Representationcited by 392
  11. 2019The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervisioncited by 372
  12. 2021Deep ViT Features as Dense Visual Descriptorscited by 330
  13. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 305
  14. 2020Integrated Task and Motion Planningcited by 290
  15. 2023TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinementcited by 287
  16. 2023PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Trackingcited by 276
  17. 2024RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Roboticscited by 251
  18. 2018Graph networks as learnable physics engines for inference and controlcited by 244
  19. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionalitycited by 243
  20. 2018Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objectscited by 241
  21. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 226
  22. 2023RVT: Robotic View Transformer for 3D Object Manipulationcited by 225
  23. 2018Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluidscited by 220
  24. 2022Particle Video Revisited: Tracking Through Occlusions Using Point Trajectoriescited by 217
  25. 2023Foundation Models in Robotics: Applications, Challenges, and the Futurecited by 216
  26. 2023Any-point Trajectory Modeling for Policy Learningcited by 215
  27. 2021Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulationcited by 209
  28. 2022AnyGrasp: Robust and Efficient Grasp Perception in Spatial and Temporal Domainscited by 208
  29. 2022MegaPose: 6D Pose Estimation of Novel Objects via Render & Comparecited by 201
  30. 2023Tracking Everything Everywhere All at Oncecited by 193
  31. 2023Putting the Object Back into Video Object Segmentationcited by 188
  32. 2024SpatialTracker: Tracking Any 2D Pixels in 3D Spacecited by 181
  33. 2023Physically Grounded Vision-Language Models for Robotic Manipulationcited by 165
  34. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 164
  35. 2023Video Language Planningcited by 162
  36. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planningcited by 161
  37. 2019kPAM: KeyPoint Affordances for Category-Level Robotic Manipulationcited by 153
  38. 2022Deep Spectral Methods: A Surprisingly Strong Baseline for Unsupervised Semantic Segmentation and Localizationcited by 150

50 most cited of 108 All 108 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 32; 2025: 143; 2026: 140.

315 papers.

Sort:

By area:

  1. 2025FAST: Efficient Action Tokenization for Vision-Language-Action Modelscited by 665
  2. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Modelscited by 451
  3. 2024TraceVLA: Visual Trace Prompting Enhances Spatial-Temporal Awareness for Generalist Robotic Policiescited by 304
  4. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agentscited by 221
  5. 2025MolmoAct: Action Reasoning Models that can Reason in Spacecited by 197
  6. 2025HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Modelcited by 181
  7. 2024VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Taskscited by 175
  8. 2024Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Makingcited by 144
  9. 20243DSRBench: A Comprehensive 3D Spatial Reasoning Benchmarkcited by 139
  10. 2024RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Roboticscited by 136
  11. 2024AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulationcited by 125
  12. 2025RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twinscited by 121
38 more
  1. 2025OmniSpatial: Towards Comprehensive Spatial Reasoning Benchmark for Vision Language Modelscited by 114
  2. 2024GRAPE: Generalizing Robot Policy via Preference Alignmentcited by 109
  3. 2025DexGraspVLA: A Vision-Language-Action Framework Towards General Dexterous Graspingcited by 89
  4. 2025VLA-Cache: Efficient Vision-Language-Action Manipulation via Adaptive Token Cachingcited by 86
  5. 2025InternVLA-M1: A Spatially Guided Vision-Language-Action Framework for Generalist Robot Policycited by 84
  6. 2026PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulationcited by 82
  7. 2024RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twins (early version)cited by 69
  8. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 69
  9. 2025SpatialReasoner: Towards Explicit and Generalizable 3D Spatial Reasoningcited by 68
  10. 2024Code-as-Monitor: Constraint-aware Visual Programming for Reactive and Proactive Robotic Failure Detectioncited by 63
  11. 20254D-VLA: Spatiotemporal Vision-Language-Action Pretraining with Cross-Scene Calibrationcited by 56
  12. 2024GenSim2: Scaling Robot Data Generation with Multi-modal and Reasoning LLMscited by 52
  13. 2025TGRPO :Fine-tuning Vision-Language-Action Model via Trajectory-wise Group Relative Policy Optimizationcited by 49
  14. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 49
  15. 2024VidMan: Exploiting Implicit Dynamics from Video Diffusion Model for Effective Robot Manipulationcited by 46
  16. 2025SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulationcited by 46
  17. 2024RT-Affordance: Affordances are Versatile Intermediate Representations for Robot Manipulationcited by 43
  18. 2025NovaFlow: Zero-Shot Manipulation via Actionable Flow from Generated Videoscited by 40
  19. 2024VisualPredicator: Learning Abstract World Models with Neuro-Symbolic Predicates for Robot Planningcited by 39
  20. 2025Open3D-VQA: A Benchmark for Comprehensive Spatial Reasoning with Multimodal Large Language Model in Open Spacecited by 39
  21. 2025Robotic Manipulation by Imitating Generated Videos Without Physical Demonstrationscited by 38
  22. 2025Embodied-R1: Reinforced Embodied Reasoning for General Robotic Manipulationcited by 38
  23. 2024VLM See, Robot Do: Human Demo Video to Robot Action Plan via Vision Language Modelcited by 37
  24. 20253DFlowAction: Learning Cross-Embodiment Manipulation from 3D Flow World Modelcited by 37
  25. 2025VLA-OS: Structuring and Dissecting Planning Representations and Paradigms in Vision-Language-Action Modelscited by 36
  26. 2025From Seeing to Doing: Bridging Reasoning and Decision for Robotic Manipulationcited by 35
  27. 2025MoManipVLA: Transferring Vision-language-action Models for General Mobile Manipulationcited by 34
  28. 2025Exploring the Limits of Vision-Language-Action Manipulations in Cross-task Generalizationcited by 33
  29. 2026ACoT-VLA: Action Chain-of-Thought for Vision-Language-Action Modelscited by 33
  30. 2025FP3: A 3D Foundation Policy for Robotic Manipulationcited by 32
  31. 2025ZeroMimic: Distilling Robotic Manipulation Skills from Web Videoscited by 32
  32. 2025GraspGen: A Diffusion-based Framework for 6-DOF Grasping with On-Generator Trainingcited by 32
  33. 2024CoA-VLA: Improving Vision-Language-Action Models via Visual-Textual Chain-of-Affordancecited by 31
  34. 2025Fast-in-Slow: A Dual-System Foundation Model Unifying Fast Manipulation within Slow Reasoningcited by 31
  35. 2024GREAT: Geometry-Intention Collaborative Inference for Open-Vocabulary 3D Object Affordance Groundingcited by 28
  36. 2024M2Diffuser: Diffusion-based Trajectory Optimization for Mobile Manipulation in 3D Scenescited by 27
  37. 2024In-Context Learning Enables Robot Action Prediction in LLMscited by 27
  38. 2025Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Braincited by 25

50 most cited of 315 All 315 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025GeoManip: Geometric Constraints as General Interfaces for Robot Manipulation
  2. 2025OmniManip: Towards General Robotic Manipulation via Object-Centric Interaction Primitives as Spatial Constraints
  3. 2026GeoProp: Grounding Robot State in Vision for Generalist Manipulation
  4. 2023Spatial Reasoning via Deep Vision Models for Robotic Sequential Manipulation
  5. 2026VolumeDP: Modeling Volumetric Representation for Manipulation Policy Learning
  6. 2026KeyGen: Unsupervised Keypoint based Object-Centric Representations for Category-Level Policy Generalization
  7. 2026RoBoSR: Structured Scene Representations for Embodied Robotic Reasoning ↗
  8. 2025Act2Goal: From World Model To General Goal-conditioned Policy