mlsanityData to 7 Oct 2026

VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models

arXiv 2307.05973 ↗First posted 12 Jul 2023cs.RO cs.AI cs.CLcited by 733

Not on the map. Why?

Cited by 147 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models (LLMs) are shown to possess a wealth of actionable knowledge that can be extracted for robot manipulation in the form of reasoning and planning. Despite the progress, most still rely on pre-defined motion primitives to carry out the physical interactions with the environment, which remains a major bottleneck. In this work, we aim to synthesize robot trajectories, i.e., a dense sequence of 6-DoF end-effector waypoints, for a large variety of manipulation tasks given an open-set of instructions and an open-set of objects. We achieve this by first observing that LLMs excel at inferring affordances and constraints given a free-form language instruction. More importantly, by leveraging their code-writing capabilities, they can interact with a vision-language model (VLM) to compose 3D value maps to ground the knowledge into the observation space of the agent. The composed value maps are then used in a model-based planning framework to zero-shot synthesize closed-loop robot trajectories with robustness to dynamic perturbations. We further demonstrate how the proposed framework can benefit from online experiences by efficiently learning a dynamics model for scenes that involve contact-rich interactions. We present a large-scale study of the proposed method in both simulated and real-robot environments, showcasing the ability to perform a large variety of everyday manipulation tasks specified in free-form natural language. Videos and code at https://voxposer.github.io

Key facts. Cited by 733 papers in the map (data through 7 Oct 2026). Builds on 107 of its 146 references that are in the map. First posted on arXiv 12 Jul 2023.

Builds on

107 of 146 references.

References · 107

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2023Segment Anythingcited by 8,158
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  11. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  12. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
38 more
  1. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  2. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  3. 2022Emergent Abilities of Large Language Modelscited by 2,483
  4. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  5. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  6. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  7. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  8. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  9. 20163D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotationcited by 1,471
  10. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  11. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  12. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  13. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  14. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  15. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  16. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  17. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
  18. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  19. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  20. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  21. 2020SAPIEN: A SimulAted Part-based Interactive ENvironmentcited by 584
  22. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  23. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  24. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  25. 2022Visual Programming: Compositional visual reasoning without trainingcited by 507
  26. 2016Interaction Networks for Learning about Objects, Relations and Physicscited by 503
  27. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 496
  28. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  29. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  30. 2022XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Modelcited by 436
  31. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 435
  32. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  33. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
  34. 2022Visual Language Maps for Robot Navigationcited by 362
  35. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  36. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  37. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 303
  38. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 296

50 most cited of 107 All 107 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 56; 2024: 215; 2025: 212; 2026: 250.

733 papers.

Sort:

By area:

  1. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  2. 2025Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Successcited by 844
  3. 2024Executable Code Actions Elicit Better LLM Agentscited by 537
  4. 2024DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Modelscited by 482
  5. 2026World Action Models are Zero-shot Policiescited by 411
  6. 2024TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulationcited by 331
  7. 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 305
  8. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 288
  9. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  10. 2024RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Roboticscited by 242
  11. 2024ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AIcited by 216
  12. 2025Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Modelscited by 216
38 more
  1. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agentscited by 213
  2. 2023RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulationcited by 206
  3. 2023DiLu: A Knowledge-Driven Approach to Autonomous Driving with Large Language Modelscited by 204
  4. 2025DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledgecited by 203
  5. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 192
  6. 2023LanguageMPC: Large Language Models as Decision Makers for Autonomous Drivingcited by 188
  7. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMscited by 187
  8. 2025HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Modelcited by 176
  9. 2024VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Taskscited by 171
  10. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 170
  11. 2024RoboMIND: Benchmark on Multi-embodiment Intelligence Normative Data for Robot Manipulationcited by 166
  12. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planningcited by 161
  13. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learningcited by 158
  14. 2023ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulationcited by 157
  15. 2023Text2Reward: Reward Shaping with Language Models for Reinforcement Learningcited by 153
  16. 2026Advancing Open-source World Modelscited by 144
  17. 2024Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Makingcited by 142
  18. 2026Cosmos 3: Omnimodal World Models for Physical AIcited by 137
  19. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interactioncited by 133
  20. 20243DSRBench: A Comprehensive 3D Spatial Reasoning Benchmarkcited by 133
  21. 2023Large Language Models for Robotics: A Surveycited by 132
  22. 2024THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulationcited by 128
  23. 2024MOKA: Open-World Robotic Manipulation through Mark-Based Visual Promptingcited by 126
  24. 2025HAMSTER: Hierarchical Action Models For Open-World Robot Manipulationcited by 125
  25. 2025Spatial Forcing: Implicit Spatial Representation Alignment for Vision-language-action Modelcited by 125
  26. 2024Yell At Your Robot: Improving On-the-Fly from Language Correctionscited by 123
  27. 2025RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twinscited by 117
  28. 2024URDFormer: A Pipeline for Constructing Articulated Simulation Environments from Real-World Imagescited by 106
  29. 2024SG-Nav: Online 3D Scene Graph Prompting for LLM-based Zero-shot Object Navigationcited by 106
  30. 2023ADriver-I: A General World Model for Autonomous Drivingcited by 102
  31. 2024RoboMamba: Efficient Vision-Language-Action Model for Robotic Reasoning and Manipulationcited by 102
  32. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 99
  33. 20233D-GPT: Procedural 3D Modeling with Large Language Modelscited by 91
  34. 2024Large Language Models for Robotics: Opportunities, Challenges, and Perspectivescited by 88
  35. 2025DexGraspVLA: A Vision-Language-Action Framework Towards General Dexterous Graspingcited by 87
  36. 2024MapGPT: Map-Guided Prompting with Adaptive Path Planning for Vision-and-Language Navigationcited by 85
  37. 2025InternVLA-M1: A Spatially Guided Vision-Language-Action Framework for Generalist Robot Policycited by 83
  38. 2024Manipulate-Anything: Automating Real-World Robots using Vision-Language Modelscited by 82

50 most cited of 733 All 733 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Reinforced Embodied Planning with Verifiable Reward for Real-World Robotic Manipulation
  2. 2025OmniManip: Towards General Robotic Manipulation via Object-Centric Interaction Primitives as Spatial Constraints
  3. 2026BridgeVLA++: A Data-Efficient, Generalizable, and Memory-Augmented Vision-Language-Action Framework for 3D Manipulation
  4. 2026Language Movement Primitives: Grounding Language Models in Robot Motion
  5. 2025From Seeing to Doing: Bridging Reasoning and Decision for Robotic Manipulation
  6. 2025Perceiving, Reasoning, Adapting: A Dual-Layer Framework for VLM-Guided Precision Robotic Manipulation
  7. 2024GravMAD: Grounded Spatial Value Maps Guided Action Diffusion for Generalized 3D Manipulation
  8. 2025ManualVLA: A Unified VLA Model for Chain-of-Thought Manual Generation and Robotic Manipulation