mlsanityData to 7 Oct 2026

ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AI

arXiv 2410.00425 ↗First posted 1 Oct 2024cs.RO cs.AIcited by 216

Not on the map. Why?

Cited by 55 in Vision-language-action models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Simulation has enabled unprecedented compute-scalable approaches to robot learning. However, many existing simulation frameworks typically support a narrow range of scenes/tasks and lack features critical for scaling generalizable robotics and sim2real. We introduce and open source ManiSkill3, the fastest state-visual GPU parallelized robotics simulator with contact-rich physics targeting generalizable manipulation. ManiSkill3 supports GPU parallelization of many aspects including simulation+rendering, heterogeneous simulation, pointclouds/voxels visual input, and more. Simulation with rendering on ManiSkill3 can run 10-1000x faster with 2-3x less GPU memory usage than other platforms, achieving up to 30,000+ FPS in benchmarked environments due to minimal python/pytorch overhead in the system, simulation on the GPU, and the use of the SAPIEN parallel rendering system. Tasks that used to take hours to train can now take minutes. We further provide the most comprehensive range of GPU parallelized environments/tasks spanning 12 distinct domains including but not limited to mobile manipulation for tasks such as drawing, humanoids, and dextrous manipulation in realistic scenes designed by artists or real-world digital twins. In addition, millions of demonstration frames are provided from motion planning, RL, and teleoperation. ManiSkill3 also provides a comprehensive set of baselines that span popular RL and learning-from-demonstrations algorithms.

Key facts. Cited by 216 papers in the map (data through 7 Oct 2026). Builds on 41 of its 52 references that are in the map. First posted on arXiv 1 Oct 2024.

Builds on

41 of 52 references.

References · 41

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  4. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  5. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  6. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 882
  7. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  8. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  9. 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 801
  10. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  11. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  12. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
29 more
  1. 2020SAPIEN: A SimulAted Part-based Interactive ENvironmentcited by 584
  2. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  3. 2018PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understandingcited by 521
  4. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 501
  5. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 484
  6. 2024Evaluating Real-World Robot Manipulation Policies in Simulationcited by 484
  7. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 482
  8. 2021Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learningcited by 395
  9. 2024RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robotscited by 391
  10. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 375
  11. 2020dm_control: Software and Tasks for Continuous Controlcited by 368
  12. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  13. 2023ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skillscited by 313
  14. 2023MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrationscited by 308
  15. 2023Orbit: A Unified Simulation Framework for Interactive Robot Learning Environmentscited by 239
  16. 2020PettingZoo: Gym for Multi-Agent Reinforcement Learningcited by 209
  17. 2021ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrationscited by 193
  18. 2024HumanPlus: Humanoid Shadowing and Imitation from Humanscited by 164
  19. 2023Habitat Synthetic Scenes Dataset (HSSD-200): An Analysis of 3D Scene Scale and Realism Tradeoffs for ObjectGoal Navigationcited by 162
  20. 2024Open-TeleVision: Teleoperation with Immersive Active Visual Feedbackcited by 137
  21. 2022DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Realitycited by 136
  22. 2023AnyTeleop: A General Vision-Based Dexterous Robot Arm-Hand Teleoperation Systemcited by 136
  23. 2022Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learningcited by 113
  24. 2015Benchmarking in Manipulation Research: The YCB Object and Model Set and Benchmarking Protocolscited by 98
  25. 2024Bunny-VisionPro: Real-Time Bimanual Dexterous Teleoperation for Imitation Learningcited by 69
  26. 2022Approximate Convex Decomposition for 3D Meshes with Collision-Aware Concavity and Tree Searchcited by 61
  27. 2024TRANSIC: Sim-to-Real Policy Transfer by Learning from Online Correctioncited by 50
  28. 2023Finetuning Offline World Models in the Real Worldcited by 39
  29. 2023EasyHeC: Accurate and Automatic Hand-eye Calibration via Differentiable Rendering and Space Explorationcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 89; 2026: 123.

216 papers.

Sort:

By area:

  1. 2024VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Taskscited by 171
  2. 2025RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twinscited by 117
  3. 2025What Can RL Bring to VLA Generalization? An Empirical Studycited by 102
  4. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  5. 2026RoboCasa365: A Large-Scale Simulation Framework for Training and Benchmarking Generalist Robotscited by 65
  6. 2025RLinf: Flexible and Efficient Large-scale Reinforcement Learning via Macro-to-Micro Flow Transformationcited by 51
  7. 2026Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisonscited by 51
  8. 2025AutoEval: Autonomous Evaluation of Generalist Robot Manipulation Policies in the Real Worldcited by 46
  9. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 46
  10. 2026World Action Models: The Next Frontier in Embodied AIcited by 46
  11. 2026SAGE: Scalable Agentic 3D Scene Generation for Embodied AIcited by 40
  12. 2025Meta-World+: An Improved, Standardized, RL Benchmarkcited by 37
38 more
  1. 2025RoboFAC: A Comprehensive Framework for Robotic Failure Analysis and Correctioncited by 37
  2. 2025Memory, Benchmark & Robots: A Benchmark for Solving Complex Tasks with Reinforcement Learningcited by 34
  3. 2025A Survey on Efficient Vision-Language-Action Modelscited by 34
  4. 2025Don't Blind Your VLA: Aligning Visual Representations for OOD Generalizationcited by 34
  5. 2025FailSafe: Reasoning and Recovery from Failures in Vision-Language-Action Modelscited by 33
  6. 2025FP3: A 3D Foundation Policy for Robotic Manipulationcited by 31
  7. 2026ACoT-VLA: Action Chain-of-Thought for Vision-Language-Action Modelscited by 30
  8. 2024Diff-DAgger: Uncertainty Estimation with Diffusion Policy for Robotic Manipulationcited by 29
  9. 2025Real-to-Sim Robot Policy Evaluation with Gaussian Splatting Simulation of Soft-Body Interactionscited by 27
  10. 2025RLinf-VLA: A Unified and Efficient Framework for Reinforcement Learning of Vision-Language-Action Modelscited by 26
  11. 2026RLDX-1 Technical Reportcited by 25
  12. 2025ManipBench: Benchmarking Vision-Language Models for Low-Level Robot Manipulationcited by 24
  13. 2025STARE-VLA: Progressive Stage-Aware Reinforcement for Fine-Tuning Vision-Language-Action Modelscited by 24
  14. 2025Dex1B: Learning with 1B Demonstrations for Dexterous Manipulationcited by 23
  15. 2025Do Vision-Language Models Have Internal World Models? Towards an Atomic Evaluationcited by 23
  16. 2025GaussGym: An open-source real-to-sim framework for learning locomotion from pixelscited by 22
  17. 2024ManiSkill-HAB: A Benchmark for Low-Level Manipulation in Home Rearrangement Taskscited by 20
  18. 2025VR-Robo: A Real-to-Sim-to-Real Framework for Visual Robot Navigation and Locomotioncited by 20
  19. 2025RLRC: Reinforcement Learning-based Recovery for Compressed Vision-Language-Action Modelscited by 20
  20. 2025Refined Policy Distillation: From VLA Generalists to RL Expertscited by 19
  21. 2025Dexterous Manipulation through Imitation Learning: A Surveycited by 18
  22. 2025VIKI-R: Coordinating Embodied Multi-Agent Cooperation via Reinforcement Learningcited by 18
  23. 2025Align-Then-stEer: Adapting the Vision-Language Action Models through Unified Latent Guidancecited by 18
  24. 2025AutoEnv: Automated Environments for Measuring Cross-Environment Agent Learningcited by 18
  25. 2026EventVLA: Event-Driven Visual Evidence Memory for Long-Horizon Vision-Language-Action Policiescited by 18
  26. 2026Show-Harness: Just a VLM Agent Can Play Robotscited by 17
  27. 2025A Survey of Robotic Navigation and Manipulation with Physics Simulators in the Era of Embodied AIcited by 16
  28. 2025A Smooth Sea Never Made a Skilled SAILOR: Robust Imitation via Learning to Searchcited by 16
  29. 2025GSWorld: Closed-Loop Photo-Realistic Simulation Suite for Robotic Manipulationcited by 16
  30. 2025π_textttRL: Online RL Fine-tuning for Flow-based Vision-Language-Action Modelscited by 16
  31. 2025MergeVLA: Cross-Skill Model Merging Toward a Generalist Vision-Language-Action Agentcited by 15
  32. 2025ReCode: Unify Plan and Action for Universal Granularity Controlcited by 14
  33. 2026On-the-Fly VLA Adaptation via Test-Time Reinforcement Learningcited by 14
  34. 2025Virtual Community: An Open World for Humans, Robots, and Societycited by 13
  35. 2025Reliable and Scalable Robot Policy Evaluation with Imperfect Simulatorscited by 13
  36. 2025Action Tokenizer Matters in In-Context Imitation Learningcited by 12
  37. 2025HumanoidGen: Data Generation for Bimanual Dexterous Manipulation via LLM Reasoningcited by 12
  38. 2025REALM: A Real-to-Sim Validated Benchmark for Generalization in Robotic Manipulationcited by 12

50 most cited of 216 All 216 (JSON)