# OCRTOC: A Cloud-Based Competition and Benchmark for Robotic Grasping and Manipulation

arXiv: https://arxiv.org/abs/2104.11446 · HTML: https://mlsanity.com/paper/2104.11446/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Apr 2021.

## Built on by (most cited first, 24 of 24)

- [Evaluating Real-World Robot Manipulation Policies in Simulation](https://mlsanity.com/paper/2405.05941/) (2024)
- [VIMA: General Robot Manipulation with Multimodal Prompts](https://mlsanity.com/paper/2210.03094/) (2022)
- [iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Tasks](https://mlsanity.com/paper/2108.03272/) (2021)
- [DexArt: Benchmarking Generalizable Dexterous Manipulation with Articulated Objects](https://mlsanity.com/paper/2305.05706/) (2023)
- [Train Offline, Test Online: A Real Robot Learning Benchmark](https://mlsanity.com/paper/2306.00942/) (2023)
- [IFOR: Iterative Flow Minimization for Robotic Object Rearrangement](https://mlsanity.com/paper/2202.00732/) (2022)
- [A Joint Modeling of Vision-Language-Action for Target-oriented Grasping in Clutter](https://mlsanity.com/paper/2302.12610/) (2023)
- [Frame Mining: a Free Lunch for Learning Robotic Manipulation from 3D Point Clouds](https://mlsanity.com/paper/2210.07442/) (2022)
- [Semantically Grounded Object Matching for Robust Robotic Scene Rearrangement](https://mlsanity.com/paper/2111.07975/) (2021)
- [Reliable and Scalable Robot Policy Evaluation with Imperfect Simulators](https://mlsanity.com/paper/2510.04354/) (2025)
- [MetaGraspNet: A Large-Scale Benchmark Dataset for Scene-Aware Ambidextrous Bin Picking via Physics-based Metaverse Synthesis](https://mlsanity.com/paper/2208.03963/) (2022)
- [Unseen Object 6D Pose Estimation: A Benchmark and Baselines](https://mlsanity.com/paper/2206.11808/) (2022)
- [LLM-enhanced Scene Graph Learning for Household Rearrangement](https://mlsanity.com/paper/2408.12093/) (2024)
- [Research Challenges and Progress in Robotic Grasping and Manipulation Competitions](https://mlsanity.com/paper/2108.01483/) (2021)
- [SCENEREPLICA: Benchmarking Real-World Robot Manipulation by Creating Replicable Scenes](https://mlsanity.com/paper/2306.15620/) (2023)
- [Sim2Real Object-Centric Keypoint Detection and Description](https://mlsanity.com/paper/2202.00448/) (2022)
- [Planning Irregular Object Packing via Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2211.09382/) (2022)
- [Neural Robot Dynamics](https://mlsanity.com/paper/2508.15755/) (2025)
- [Grasp, See, and Place: Efficient Unknown Object Rearrangement with Policy Structure Prior](https://mlsanity.com/paper/2402.15402/) (2024)
- [Rethinking the Intermediate Features in Adversarial Attacks: Misleading Robotic Models via Adversarial Distillation](https://mlsanity.com/paper/2411.15222/) (2024)
- [PhAIL: A Real-Robot VLA Benchmark and Distributional Methodology](https://mlsanity.com/paper/2605.29710/) (2026)
- [Category-level Shape Estimation for Densely Cluttered Objects](https://mlsanity.com/paper/2302.11983/) (2023)
- [NormNet: Scale Normalization for 6D Pose Estimation in Stacked Scenarios](https://mlsanity.com/paper/2311.09269/) (2023)
- [Beyond Visual Grasping: Benchmarking Complex Grasping from Detection to Execution](https://mlsanity.com/paper/2607.14341/) (2026)

## Abstract (arXiv)

In this paper, we propose a cloud-based benchmark for robotic grasping and manipulation, called the OCRTOC benchmark. The benchmark focuses on the object rearrangement problem, specifically table organization tasks. We provide a set of identical real robot setups and facilitate remote experiments of standardized table organization scenarios in varying difficulties. In this workflow, users upload their solutions to our remote server and their code is executed on the real robot setups and scored automatically. After each execution, the OCRTOC team resets the experimental setup manually. We also provide a simulation environment that researchers can use to develop and test their solutions. With the OCRTOC benchmark, we aim to lower the barrier of conducting reproducible research on robotic grasping and manipulation and accelerate progress in this field. Executing standardized scenarios on identical real robot setups allows us to quantify algorithm performances and achieve fair comparisons. Using this benchmark we held a competition in the 2020 International Conference on Intelligence Robots and Systems (IROS 2020). In total, 59 teams took part in this competition worldwide. We present the results and our observations of the 2020 competition, and discuss our adjustments and improvements for the upcoming OCRTOC 2021 competition. The homepage of the OCRTOC competition is www.ocrtoc.org, and the OCRTOC software package is available at https://github.com/OCRTOC/OCRTOC_software_package.