mlsanityData to 7 Oct 2026

Manipulate-Anything: Automating Real-World Robots using Vision-Language Models

arXiv 2406.18915 ↗First posted 27 Jun 2024cs.RO cs.CVcited by 82

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Large-scale endeavors like and widespread community efforts such as Open-X-Embodiment have contributed to growing the scale of robot demonstration data. However, there is still an opportunity to improve the quality, quantity, and diversity of robot demonstration data. Although vision-language models have been shown to automatically generate demonstration data, their utility has been limited to environments with privileged state information, they require hand-designed skills, and are limited to interactions with few object instances. We propose Manipulate-Anything, a scalable automated generation method for real-world robotic manipulation. Unlike prior work, our method can operate in real-world environments without any privileged state information, hand-designed skills, and can manipulate any static object. We evaluate our method using two setups. First, Manipulate-Anything successfully generates trajectories for all 7 real-world and 14 simulation tasks, significantly outperforming existing methods like VoxPoser. Second, Manipulate-Anything's demonstrations can train more robust behavior cloning policies than training with human demonstrations, or from data generated by VoxPoser, Scaling-up, and Code-As-Policies. We believe Manipulate-Anything can be a scalable method for both generating data for robotics and solving novel tasks in a zero-shot setting. Project page: https://robot-ma.github.io/.

Key facts. Cited by 82 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 48 of its 63 references that are in the map. First posted on arXiv 27 Jun 2024.

Builds on

48 of 63 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024RoboPoint: A Vision-Language Model for Spatial Affordance Prediction for Robotics15 Jun
  2. 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedback6 Feb

Other references · 46

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  7. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  8. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  9. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  10. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  11. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  12. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
34 more
  1. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  2. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  3. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  4. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  5. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  6. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  7. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
  8. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  9. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  10. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  11. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  12. 2024Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robotscited by 373
  13. 2020Integrated Task and Motion Planningcited by 281
  14. 2023MEDITRON-70B: Scaling Medical Pretraining for Large Language Modelscited by 272
  15. 2023Demystifying CLIP Datacited by 265
  16. 2023RVT: Robotic View Transformer for 3D Object Manipulationcited by 224
  17. 2023Text2Motion: From Natural Language Instructions to Feasible Planscited by 223
  18. 2023RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulationcited by 206
  19. 2023Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisitioncited by 156
  20. 2024DexCap: Scalable and Portable Mocap Data Collection System for Dexterous Manipulationcited by 149
  21. 2022Data Determines Distributional Robustness in Contrastive Language Image Pre-training (CLIP)cited by 137
  22. 2024RVT-2: Learning Precise Manipulation from Few Demonstrationscited by 135
  23. 2023GenSim: Generating Robotic Simulation Tasks via Large Language Modelscited by 120
  24. 2022Quality Not Quantity: On the Interaction between Dataset Design and Robustness of CLIPcited by 97
  25. 2024CoPa: General Robotic Manipulation through Spatial Constraints of Parts with Foundation Modelscited by 97
  26. 2023Improving Multimodal Datasets with Image Captioningcited by 93
  27. 2023Bootstrap Your Own Skills: Learning to Solve New Tasks with Large Language Model Guidancecited by 85
  28. 2024AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agentscited by 83
  29. 2024No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performancecited by 75
  30. 2023Learning Vision from Models Rivals Learning Vision from Datacited by 71
  31. 2023AR2-D2:Training a Robot Without a Robotcited by 26
  32. 2023On the Connection between Pre-training Data Diversity and Fine-tuning Robustnesscited by 25
  33. 2023MVTrans: Multi-View Perception of Transparent Objectscited by 14
  34. 2021A Benchmark for Modeling Violation-of-Expectation in Physical Reasoning Across Event Categoriescited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 16; 2025: 39; 2026: 27.

82 papers.

Sort:

By area:

  1. 2025π0.5: a Vision-Language-Action Model with Open-World Generalizationcited by 1,653
  2. 2025Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Successcited by 844
  3. 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 305
  4. 2025HAMSTER: Hierarchical Action Models For Open-World Robot Manipulationcited by 125
  5. 2026StarVLA: A Lego-like Codebase for Vision-Language-Action Model Developingcited by 103
  6. 2025Motion Tracks: A Unified Representation for Human-Robot Transfer in Few-Shot Imitation Learningcited by 57
  7. 2024GenSim2: Scaling Robot Data Generation with Multi-modal and Reasoning LLMscited by 48
  8. 2025Long-VLA: Unleashing Long-Horizon Capability of Vision Language Action Model for Robot Manipulationcited by 48
  9. 2025SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulationcited by 46
  10. 2024RT-Affordance: Affordances are Versatile Intermediate Representations for Robot Manipulationcited by 40
  11. 2025Exploring the Limits of Vision-Language-Action Manipulations in Cross-task Generalizationcited by 31
  12. 2025PointArena: Probing Multimodal Grounding Through Language-Guided Pointingcited by 30
38 more
  1. 2025Think Twice, Act Once: Token-Aware Compression and Action Reuse for Efficient Inference in Vision-Language-Action Modelscited by 30
  2. 2026ACoT-VLA: Action Chain-of-Thought for Vision-Language-Action Modelscited by 30
  3. 2024CoA-VLA: Improving Vision-Language-Action Models via Visual-Textual Chain-of-Affordancecited by 29
  4. 2024In-Context Learning Enables Robot Action Prediction in LLMscited by 25
  5. 2025UAD: Unsupervised Affordance Distillation for Generalization in Robotic Manipulationcited by 23
  6. 2025Robix: A Unified Model for Robot Interaction, Reasoning and Planningcited by 23
  7. 2024RoboMatrix: A Skill-centric Hierarchical Framework for Scalable Robot Task Planning and Execution in Open-Worldcited by 22
  8. 2024MALMM: Multi-Agent Large Language Models for Zero-Shot Robotics Manipulationcited by 18
  9. 2025Seeing to Act, Prompting to Specify: A Bayesian Factorization of Vision Language Action Policycited by 17
  10. 2026Show-Harness: Just a VLM Agent Can Play Robotscited by 17
  11. 2024The One RING: a Robotic Indoor Navigation Generalistcited by 15
  12. 2025MMPB: It's Time for Multi-Modal Personalizationcited by 14
  13. 2025MoMa-Kitchen: A 100K+ Benchmark for Affordance-Grounded Last-Mile Navigation in Mobile Manipulationcited by 12
  14. 2026WAM4D: Fast 4D World Action Model via Spatial Register Tokenscited by 12
  15. 2024From Pixels to Predicates: Learning Symbolic World Models via Pretrained Vision-Language Modelscited by 11
  16. 2025Enhancing Autonomous Driving Systems with On-Board Deployed Large Language Modelscited by 11
  17. 2025Point-It-Out: Benchmarking Embodied Reasoning for Vision Language Models in Multi-Stage Visual Groundingcited by 10
  18. 2024Open-World Task and Motion Planning via Vision-Language Model Generated Constraintscited by 9
  19. 2024Helpful DoggyBot: Open-World Object Fetching using Legged Robots and Vision-Language Modelscited by 8
  20. 2025CodeDiffuser: Attention-Enhanced Diffusion Policy via VLM-Generated Code for Instruction Ambiguitycited by 8
  21. 2025End-to-End Dexterous Arm-Hand VLA Policies via Shared Autonomy: VR Teleoperation Augmented by Autonomous Hand VLA Policy for Efficient Data Collectioncited by 8
  22. 2025Guardian: Detecting Robotic Planning and Execution Errors with Vision-Language Modelscited by 8
  23. 2025SIMPACT: Simulation-Enabled Action Planning using Vision-Language Modelscited by 7
  24. 2026Guava: Distilling Frontier VLMs into a Compact Agent through a Robotic Manipulation Harnesscited by 7
  25. 2026PLanAR: Planning-Language-Grounded Agentic Reasoning for Robot Manipulationcited by 6
  26. 2026From Reaction to Anticipation: Proactive Failure Recovery through Agentic Task Graph for Robotic Manipulationcited by 6
  27. 2026ETA: A New Agentic Paradigm for Embodied Taskscited by 6
  28. 2024Select2Plan: Training-Free ICL-Based Planning through VQA and Memory Retrievalcited by 5
  29. 2025The Safety Challenge of World Models for Embodied AI Agents: A Reviewcited by 5
  30. 2026GeneralVLA: Generalizable Vision-Language-Action Models with Knowledge-Guided Trajectory Planningcited by 5
  31. 2025A Real-to-Sim-to-Real Approach to Robotic Manipulation with VLM-Generated Iterative Keypoint Rewardscited by 4
  32. 2026Dual Latent Memory in Vision-Language-Action Models for Robotic Manipulationcited by 4
  33. 2024Task-oriented Robotic Manipulation with Vision Language Modelscited by 3
  34. 2025Executable Analytic Concepts as the Missing Link Between VLM Insight and Precise Manipulationcited by 3
  35. 2025ManiAgent: An Agentic Framework for General Robotic Manipulationcited by 3
  36. 2024The State of Robot Motion Generationcited by 2
  37. 2025Cache-of-Thought: Master-Apprentice Framework for Cost-Effective Vision Language Model Reasoningcited by 2
  38. 2025EMMOE: A Comprehensive Benchmark for Embodied Mobile Manipulation in Open Environmentscited by 2

50 most cited of 82 All 82 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026SMART: Zero-Shot Sim-to-Real Articulated Object Manipulation via Large-Scale Synthetic Pretraining
  2. 2026Grounding Sim-to-Real Generalization in Robotic Manipulation: An Empirical Study with Vision-Language-Action Models
  3. 2025REALM: A Real-to-Sim Validated Benchmark for Generalization in Robotic Manipulation
  4. 2026DREAM: Deployment-Time Demonstration Generation via Real-to-Sim for Scalable Policy Adaptation
  5. 2026PointCast: One World Model for Rigid, Articulated, and Deformable Object Manipulation
  6. 2024RoboScript: Code Generation for Free-Form Manipulation Tasks across Real and Simulation
  7. 2026ManiSkillFormer: Demonstration-Free Compositional Manipulation via Geometric Contracts and Agentic Skill Graph
  8. 2026AXIS: A Growable Community-Driven Data Engine for Scalable Robot Manipulation