mlsanityData to 14 Oct 2026

ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrations

arXiv 2107.14483 ↗First posted 30 Jul 2021cs.LG cs.AI cs.CVcited by 197

Not on the map. Why?

Cited by 40 in Vision-language-action models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Object manipulation from 3D visual inputs poses many challenges on building generalizable perception and policy models. However, 3D assets in existing benchmarks mostly lack the diversity of 3D shapes that align with real-world intra-class complexity in topology and geometry. Here we propose SAPIEN Manipulation Skill Benchmark (ManiSkill) to benchmark manipulation skills over diverse objects in a full-physics simulator. 3D assets in ManiSkill include large intra-class topological and geometric variations. Tasks are carefully chosen to cover distinct types of manipulation challenges. Latest progress in 3D vision also makes us believe that we should customize the benchmark so that the challenge is inviting to researchers working on 3D deep learning. To this end, we simulate a moving panoramic camera that returns ego-centric point clouds or RGB-D images. In addition, we would like ManiSkill to serve a broad set of researchers interested in manipulation research. Besides supporting the learning of policies from interactions, we also support learning-from-demonstrations (LfD) methods, by providing a large number of high-quality demonstrations (~36,000 successful trajectories, ~1.5M point cloud/RGB-D frames in total). We provide baselines using 3D deep learning and LfD algorithms. All code of our benchmark (simulator, environment, SDK, and baselines) is open-sourced, and a challenge facing interdisciplinary researchers will be held based on the benchmark.

Key facts. Cited by 197 papers in the map (data through 14 Oct 2026). Builds on 61 of its 79 references that are in the map. First posted on arXiv 30 Jul 2021.

Builds on

61 of 79 references.

References · 61

  1. 2017Attention Is All You Needcited by 60,361
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
  4. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  5. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,897
  6. 2017CARLA: An Open Urban Driving Simulatorcited by 2,449
  7. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  8. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  9. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  10. 2016OpenAI Gymcited by 1,736
  11. 2016Generative Adversarial Imitation Learningcited by 1,597
  12. 2020Point Transformercited by 1,470
38 more
  1. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  2. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  3. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  4. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,208
  5. 2019KPConv: Flexible and Deformable Convolution for Point Cloudscited by 1,205
  6. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  7. 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
  8. 2017Hindsight Experience Replaycited by 1,136
  9. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  10. 2015End-to-End Training of Deep Visuomotor Policiescited by 875
  11. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 869
  12. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
  13. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  14. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 689
  15. 2019Deep Hough Voting for 3D Object Detection in Point Cloudscited by 648
  16. 2020SAPIEN: A SimulAted Part-based Interactive ENvironmentcited by 595
  17. 2018PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understandingcited by 526
  18. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 511
  19. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 481
  20. 2018Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Drivingcited by 457
  21. 2018VirtualHome: Simulating Household Activities via Programscited by 415
  22. 2020dm_control: Software and Tasks for Continuous Controlcited by 375
  23. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 344
  24. 2016Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collectioncited by 316
  25. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 270
  26. 2017Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metricscited by 264
  27. 2020ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulationcited by 250
  28. 20196-DOF GraspNet: Variational Grasp Generation for Object Manipulationcited by 242
  29. 2017Submanifold Sparse Convolutional Networkscited by 223
  30. 2019Category-Level Articulated Object Pose Estimationcited by 159
  31. 2017Grasp Pose Detection in Point Cloudscited by 111
  32. 2018Building Generalizable Agents with a Realistic and Rich 3D Environmentcited by 109
  33. 2021ManipulaTHOR: A Framework for Visual Object Manipulationcited by 102
  34. 2019Interactive Gibson Benchmark (iGibson 0.5): A Benchmark for Interactive Navigation in Cluttered Environmentscited by 90
  35. 2018PointNetGPD: Detecting Grasp Configurations from Point Setscited by 80
  36. 2019State Alignment-based Imitation Learningcited by 73
  37. 2021Asymmetric self-play for automatic goal discovery in robotic manipulationcited by 72
  38. 2019KETO: Learning Keypoint Representations for Tool Manipulationcited by 69

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 19; 2023: 29; 2024: 38; 2025: 48; 2026: 63.

197 papers.

Sort:

By area:

  1. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,410
  2. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 657
  3. 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 495
  4. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 380
  5. 2022HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interactioncited by 327
  6. 2023ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skillscited by 327
  7. 2023MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrationscited by 327
  8. 2025LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Modelscited by 303
  9. 2023Diffusion-based Generation, Optimization, and Planning in 3D Scenescited by 239
  10. 2024ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AIcited by 229
  11. 2023Habitat 3.0: A Co-Habitat for Humans, Avatars and Robotscited by 228
  12. 2023RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulationcited by 218
38 more
  1. 2024VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Taskscited by 175
  2. 2024Agent AI: Surveying the Horizons of Multimodal Interactioncited by 170
  3. 2024BEHAVIOR-1K: A Human-Centered, Embodied AI Benchmark with 1,000 Everyday Activities and Realistic Simulationcited by 148
  4. 2024Scaling Proprioceptive-Visual Learning with Heterogeneous Pre-trained Transformerscited by 148
  5. 2024Data Scaling Laws in Imitation Learning for Robotic Manipulationcited by 145
  6. 2022GAPartNet: Cross-Category Domain-Generalizable Object Perception and Manipulation via Generalizable and Actionable Partscited by 139
  7. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 133
  8. 2024PhyScene: Physically Interactable 3D Scene Synthesis for Embodied AIcited by 125
  9. 2023UniDexGrasp: Universal Robotic Dexterous Grasping via Learning Diverse Proposal Generation and Goal-Conditioned Policycited by 124
  10. 2023UniDexGrasp++: Improving Dexterous Grasping Policy Learning via Geometry-aware Curriculum and Iterative Generalist-Specialist Learningcited by 107
  11. 2023HomeRobot: Open-Vocabulary Mobile Manipulationcited by 106
  12. 2022DexPoint: Generalizable Point Cloud Reinforcement Learning for Sim-to-Real Dexterous Manipulationcited by 84
  13. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 84
  14. 2025Dita: Scaling Diffusion Transformer for Generalist Vision-Language-Action Policycited by 74
  15. 2024Policy Decorator: Model-Agnostic Online Refinement for Large Policy Modelcited by 70
  16. 2025VQ-VLA: Improving Vision-Language-Action Models via Scaling Vector-Quantized Action Tokenizerscited by 70
  17. 2025GeoVLA: Empowering 3D Representations in Vision-Language-Action Modelscited by 70
  18. 2023ARNOLD: A Benchmark for Language-Grounded Task Learning With Continuous States in Realistic 3D Scenescited by 69
  19. 2024SplatSim: Zero-Shot Sim2Real Transfer of RGB Manipulation Policies Using Gaussian Splattingcited by 69
  20. 2025WorldEval: World Model as Real-World Robot Policies Evaluatorcited by 68
  21. 2023DexArt: Benchmarking Generalizable Dexterous Manipulation with Articulated Objectscited by 66
  22. 2022End-to-End Affordance Learning for Robotic Manipulationcited by 57
  23. 2025RoboVerse: Towards a Unified Platform, Dataset and Benchmark for Scalable and Generalizable Robot Learningcited by 54
  24. 2024Towards Diverse Behaviors: A Benchmark for Imitation Learning with Human Demonstrationscited by 53
  25. 2022Learning Generalizable Dexterous Manipulation from Human Grasp Affordancecited by 51
  26. 2022Domain Randomization-Enhanced Depth Simulation and Restoration for Perceiving and Grasping Specular and Transparent Objectscited by 50
  27. 2026World Action Models: The Next Frontier in Embodied AIcited by 50
  28. 2025AutoEval: Autonomous Evaluation of Generalist Robot Manipulation Policies in the Real Worldcited by 48
  29. 2022Multi-skill Mobile Manipulation for Object Rearrangementcited by 46
  30. 20223D-Aware Object Goal Navigation via Simultaneous Exploration and Identificationcited by 46
  31. 2023Neural Volumetric Memory for Visual Locomotion Controlcited by 44
  32. 2023PartManip: Learning Cross-Category Generalizable Part Manipulation Policy from Point Cloud Observationscited by 42
  33. 2024Diffusion Transformer Policycited by 41
  34. 2024Diff-DAgger: Uncertainty Estimation with Diffusion Policy for Robotic Manipulationcited by 33
  35. 2025Exploring the Limits of Vision-Language-Action Manipulations in Cross-task Generalizationcited by 33
  36. 2022Frame Mining: a Free Lunch for Learning Robotic Manipulation from 3D Point Cloudscited by 32
  37. 2025VLA-Arena: An Open-Source Framework for Benchmarking Vision-Language-Action Modelscited by 31
  38. 2024BiGym: A Demo-Driven Mobile Bi-Manual Manipulation Benchmarkcited by 30

50 most cited of 197 All 197 (JSON)