# RB2: Robotic Manipulation Benchmarking with a Twist

arXiv: https://arxiv.org/abs/2203.08098 · HTML: https://mlsanity.com/paper/2203.08098/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Mar 2022.

## Built on by (most cited first, 23 of 23)

- [Evaluating Real-World Robot Manipulation Policies in Simulation](https://mlsanity.com/paper/2405.05941/) (2024)
- [MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations](https://mlsanity.com/paper/2310.17596/) (2023)
- [RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunking](https://mlsanity.com/paper/2309.01918/) (2023)
- [Human-to-Robot Imitation in the Wild](https://mlsanity.com/paper/2207.09450/) (2022)
- [VideoDex: Learning Dexterity from Internet Videos](https://mlsanity.com/paper/2212.04498/) (2022)
- [HomeRobot: Open-Vocabulary Mobile Manipulation](https://mlsanity.com/paper/2306.11565/) (2023)
- [LEAP Hand: Low-Cost, Efficient, and Anthropomorphic Hand for Robot Learning](https://mlsanity.com/paper/2309.06440/) (2023)
- [DexArt: Benchmarking Generalizable Dexterous Manipulation with Articulated Objects](https://mlsanity.com/paper/2305.05706/) (2023)
- [RoboArena: Distributed Real-World Evaluation of Generalist Robot Policies](https://mlsanity.com/paper/2506.18123/) (2025)
- [Train Offline, Test Online: A Real Robot Learning Benchmark](https://mlsanity.com/paper/2306.00942/) (2023)
- [HRP: Human Affordances for Robotic Pre-Training](https://mlsanity.com/paper/2407.18911/) (2024)
- [An Unbiased Look at Datasets for Visuo-Motor Pre-Training](https://mlsanity.com/paper/2310.09289/) (2023)
- [PerAct2: Benchmarking and Learning for Robotic Bimanual Manipulation Tasks](https://mlsanity.com/paper/2407.00278/) (2024)
- [RoboEval: Where Robotic Manipulation Meets Structured and Scalable Evaluation](https://mlsanity.com/paper/2507.00435/) (2025)
- [Real-to-Sim Robot Policy Evaluation with Gaussian Splatting Simulation of Soft-Body Interactions](https://mlsanity.com/paper/2511.04665/) (2025)
- [Real World Offline Reinforcement Learning with Realistic Data Source](https://mlsanity.com/paper/2210.06479/) (2022)
- [Learning Precise, Contact-Rich Manipulation through Uncalibrated Tactile Skins](https://mlsanity.com/paper/2410.17246/) (2024)
- [SCENEREPLICA: Benchmarking Real-World Robot Manipulation by Creating Replicable Scenes](https://mlsanity.com/paper/2306.15620/) (2023)
- [Multimodal and Force-Matched Imitation Learning with a See-Through Visuotactile Sensor](https://mlsanity.com/paper/2311.01248/) (2023)
- [Towards Open-World Mobile Manipulation in Homes: Lessons from the Neurips 2023 HomeRobot Open Vocabulary Mobile Manipulation Challenge](https://mlsanity.com/paper/2407.06939/) (2024)
- [RMBench: Benchmarking Deep Reinforcement Learning for Robotic Manipulator Control](https://mlsanity.com/paper/2210.11262/) (2022)
- [TeleOpBench: A Simulator-Centric Benchmark for Dual-Arm Dexterous Teleoperation](https://mlsanity.com/paper/2505.12748/) (2025)
- [Distributed Dexterous Manipulation with Spatially Conditioned Multi-Agent Transformers](https://mlsanity.com/paper/2609.06930/) (2026)

## Abstract (arXiv)

Benchmarks offer a scientific way to compare algorithms using objective performance metrics. Good benchmarks have two features: (a) they should be widely useful for many research groups; (b) and they should produce reproducible findings. In robotic manipulation research, there is a trade-off between reproducibility and broad accessibility. If the benchmark is kept restrictive (fixed hardware, objects), the numbers are reproducible but the setup becomes less general. On the other hand, a benchmark could be a loose set of protocols (e.g. object sets) but the underlying variation in setups make the results non-reproducible. In this paper, we re-imagine benchmarking for robotic manipulation as state-of-the-art algorithmic implementations, alongside the usual set of tasks and experimental protocols. The added baseline implementations will provide a way to easily recreate SOTA numbers in a new local robotic setup, thus providing credible relative rankings between existing approaches and new work. However, these local rankings could vary between different setups. To resolve this issue, we build a mechanism for pooling experimental data between labs, and thus we establish a single global ranking for existing (and proposed) SOTA algorithms. Our benchmark, called Ranking-Based Robotics Benchmark (RB2), is evaluated on tasks that are inspired from clinically validated Southampton Hand Assessment Procedures. Our benchmark was run across two different labs and reveals several surprising findings. For example, extremely simple baselines like open-loop behavior cloning, outperform more complicated models (e.g. closed loop, RNN, Offline-RL, etc.) that are preferred by the field. We hope our fellow researchers will use RB2 to improve their research's quality and rigor.