# The ACRV Picking Benchmark (APB): A Robotic Shelf Picking Benchmark to Foster Reproducible Research

arXiv: https://arxiv.org/abs/1609.05258 · HTML: https://mlsanity.com/paper/1609.05258/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Sep 2016.

## Built on by (most cited first, 11 of 11)

- [AutoEval: Autonomous Evaluation of Generalist Robot Manipulation Policies in the Real World](https://mlsanity.com/paper/2503.24278/) (2025)
- [ARMBench: An Object-centric Benchmark Dataset for Robotic Manipulation](https://mlsanity.com/paper/2303.16382/) (2023)
- [Simitate: A Hybrid Imitation Learning Benchmark](https://mlsanity.com/paper/1905.06002/) (2019)
- [Research Challenges and Progress in Robotic Grasping and Manipulation Competitions](https://mlsanity.com/paper/2108.01483/) (2021)
- [GKNet: grasp keypoint network for grasp candidates detection](https://mlsanity.com/paper/2106.08497/) (2021)
- [Reproducibility of Machine Learning: Terminology, Recommendations and Open Issues](https://mlsanity.com/paper/2302.12691/) (2023)
- [M3Bench: Benchmarking Whole-body Motion Generation for Mobile Manipulation in 3D Scenes](https://mlsanity.com/paper/2410.06678/) (2024)
- [ALRM: Agentic LLM for Robotic Manipulation](https://mlsanity.com/paper/2601.19510/) (2026)
- [Improving the Successful Robotic Grasp Detection Using Convolutional Neural Networks](https://mlsanity.com/paper/2403.05211/) (2024)
- [Attention-Guided Integration of CLIP and SAM for Precise Object Masking in Robotic Manipulation](https://mlsanity.com/paper/2502.18842/) (2025)
- [Australian Supermarket Object Set (ASOS): A Benchmark Dataset of Physical Objects and 3D Models for Robotics and Computer Vision](https://mlsanity.com/paper/2509.09720/) (2025)

## Abstract (arXiv)

Robotic challenges like the Amazon Picking Challenge (APC) or the DARPA Challenges are an established and important way to drive scientific progress. They make research comparable on a well-defined benchmark with equal test conditions for all participants. However, such challenge events occur only occasionally, are limited to a small number of contestants, and the test conditions are very difficult to replicate after the main event. We present a new physical benchmark challenge for robotic picking: the ACRV Picking Benchmark (APB). Designed to be reproducible, it consists of a set of 42 common objects, a widely available shelf, and exact guidelines for object arrangement using stencils. A well-defined evaluation protocol enables the comparison of complete robotic systems -- including perception and manipulation -- instead of sub-systems only. Our paper also describes and reports results achieved by an open baseline system based on a Baxter robot.