mlsanityData to 7 Oct 2026

Local Policies Enable Zero-shot Long-horizon Manipulation

arXiv 2410.22332 ↗First posted 29 Oct 2024cs.RO cs.CV cs.LGcited by 21

Not on the map. Why?

Abstract (arXiv)

Sim2real for robotic manipulation is difficult due to the challenges of simulating complex contacts and generating realistic task distributions. To tackle the latter problem, we introduce ManipGen, which leverages a new class of policies for sim2real transfer: local policies. Locality enables a variety of appealing properties including invariances to absolute robot and object pose, skill ordering, and global scene configuration. We combine these policies with foundation models for vision, language and motion planning and demonstrate SOTA zero-shot performance of our method to Robosuite benchmark tasks in simulation (97%). We transfer our local policies from simulation to reality and observe they can solve unseen long-horizon manipulation tasks with up to 8 stages with significant pose, object and scene configuration variation. ManipGen outperforms SOTA approaches such as SayCan, OpenVLA, LLMTrajGen and VoxPoser across 50 real-world manipulation tasks by 36%, 76%, 62% and 60% respectively. Video results at https://mihdalal.github.io/manipgen/

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 63 of its 83 references that are in the map. First posted on arXiv 29 Oct 2024.

Builds on

63 of 83 references.

References · 63

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  5. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  6. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  7. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  8. 2024Depth Anything: Unleashing the Power of Large-Scale Unlabeled Datacited by 1,346
  9. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  10. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  11. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  12. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 914
38 more
  1. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 882
  2. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  3. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  4. 2019Solving Rubik's Cube with a Robot Handcited by 699
  5. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  6. 2018Learning Dexterous In-Hand Manipulationcited by 643
  7. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 634
  8. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
  9. 2018PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understandingcited by 521
  10. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 496
  11. 2020Learning Quadrupedal Locomotion over Challenging Terraincited by 494
  12. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 484
  13. 2021RMA: Rapid Motor Adaptation for Legged Robotscited by 430
  14. 2024Depth Pro: Sharp Monocular Metric Depth in Less Than a Secondcited by 395
  15. 2024Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperationcited by 377
  16. 2024Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robotscited by 373
  17. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  18. 2021Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasetscited by 248
  19. 2018HG-DAgger: Interactive Imitation Learning with Human Expertscited by 245
  20. 2023Text2Motion: From Natural Language Instructions to Feasible Planscited by 223
  21. 2023TidyBot: Personalized Robot Assistance with Large Language Modelscited by 219
  22. 2021Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenescited by 213
  23. 2015Deep Spatial Autoencoders for Visuomotor Learningcited by 149
  24. 2018PDDLStream: Integrating Symbolic Planners and Blackbox Samplers via Optimistic Adaptive Planningcited by 147
  25. 2022DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Realitycited by 136
  26. 2023FurnitureBench: Reproducible Real-World Benchmark for Long-Horizon Complex Manipulationcited by 132
  27. 2022Legged Locomotion in Challenging Terrains using Egocentric Visioncited by 131
  28. 2023Extreme Parkour with Legged Robotscited by 120
  29. 2023UniDexGrasp: Universal Robotic Dexterous Grasping via Learning Diverse Proposal Generation and Goal-Conditioned Policycited by 117
  30. 2023Robot Parkour Learningcited by 115
  31. 2024Learning Human-to-Humanoid Real-Time Whole-Body Teleoperationcited by 112
  32. 2023ANYmal Parkour: Learning Agile Navigation for Quadrupedal Robotscited by 105
  33. 2020Human-in-the-Loop Imitation Learning using Remote Teleoperationcited by 103
  34. 2024CoPa: General Robotic Manipulation through Spatial Constraints of Parts with Foundation Modelscited by 97
  35. 2023Bootstrap Your Own Skills: Learning to Solve New Tasks with Large Language Model Guidancecited by 85
  36. 2021Accelerating Robotic Reinforcement Learning via Parameterized Action Primitivescited by 63
  37. 2023Imitating Task and Motion Planning with Visuomotor Transformerscited by 63
  38. 2019Online Replanning in Belief Space for Partially Observable Task and Motion Problemscited by 60

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 12; 2026: 9.

21 papers.

Sort:

By area:

  1. 2025Long-VLA: Unleashing Long-Horizon Capability of Vision Language Action Model for Robot Manipulationcited by 48
  2. 2025NovaFlow: Zero-Shot Manipulation via Actionable Flow from Generated Videoscited by 38
  3. 2025A Survey of Sim-to-Real Methods in RL: Progress, Prospects and Challenges with Foundation Modelscited by 36
  4. 2025GraspGen: A Diffusion-based Framework for 6-DOF Grasping with On-Generator Trainingcited by 32
  5. 2025The Reality Gap in Robotics: Challenges, Solutions, and Best Practicescited by 22
  6. 2026LiLo-VLA: Compositional Long-Horizon Manipulation via Linked Object-Centric Policiescited by 12
  7. 2025ArticuBot: Learning Universal Articulated Object Manipulation Policy via Large Scale Simulationcited by 10
  8. 2025Maestro: Orchestrating Robotics Modules with Vision-Language Models for Zero-Shot Generalist Robotscited by 8
  9. 2025Constraint-Preserving Data Generation for Visuomotor Policy Learningcited by 6
  10. 2026OmniGuide: Universal Guidance Fields for Enhancing Generalist Robot Policiescited by 6
  11. 2025RaySt3R: Predicting Novel Depth Maps for Zero-Shot Object Completioncited by 5
  12. 2025Learning to See and Act: Task-Aware Virtual View Exploration for Robotic Manipulationcited by 5
9 more
  1. 2025Learning Primitive Embodied World Models: Towards Scalable Robotic Learningcited by 5
  2. 2025Prepare Before You Act: Learning From Humans to Rearrange Initial Statescited by 3
  3. 2026Selective Perception for Robot: Task-Aware Attention in Multimodal VLAcited by 1
  4. 2026HERO: Learning Humanoid End-Effector Control for Visual Whole-Body Open-Vocabulary Object Graspingcited by 0
  5. 2026ADEPT: Accelerating Dexterity via Pre-Training and Post-Training using Reinforcement Learningcited by 0
  6. 2026Hierarchical Skill Retrieval for Data-Efficient Adaptation of Vision-Language-Action Modelscited by 0
  7. 2026From Rollout to Reset: A Graph-Based Harness for Autonomous Long-Horizon Manipulation Evaluationcited by 0
  8. 2026An Empirical Study on What Matters for Viewpoint-Generalizable Policies in Visual Imitation Learningcited by 0
  9. 2026ZeroBot: Learning from Scratch in Minutes with Generative Real2Simcited by 0