# The CORSMAL benchmark for the prediction of the properties of containers

arXiv: https://arxiv.org/abs/2107.12719 · HTML: https://mlsanity.com/paper/2107.12719/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 61 references that are in the map.
- First posted on arXiv 27 Jul 2021.

## Builds on (most cited first, 30 of 31)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [YOLOv4: Optimal Speed and Accuracy of Object Detection](https://mlsanity.com/paper/2004.10934/) (2020)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation](https://mlsanity.com/paper/1901.02970/) (2019)
- [BOP: Benchmark for 6D Object Pose Estimation](https://mlsanity.com/paper/1808.08319/) (2018)
- [Objectron: A Large Scale Dataset of Object-Centric Videos in the Wild with Pose Annotations](https://mlsanity.com/paper/2012.09988/) (2020)
- [ClearGrasp: 3D Shape Estimation of Transparent Objects for Manipulation](https://mlsanity.com/paper/1910.02550/) (2019)
- [Learning Canonical Shape Space for Category-Level 6D Object Pose and Size Estimation](https://mlsanity.com/paper/2001.09322/) (2020)
- [HomebrewedDB: RGB-D Dataset for 6D Pose Estimation of 3D Objects](https://mlsanity.com/paper/1904.03167/) (2019)
- [Category Level Object Pose Estimation via Neural Analysis-by-Synthesis](https://mlsanity.com/paper/2008.08145/) (2020)
- [KeyPose: Multi-View 3D Labeling and Keypoint Estimation for Transparent Objects](https://mlsanity.com/paper/1912.02805/) (2019)
- [Object Handovers: a Review for Robotics](https://mlsanity.com/paper/2007.12952/) (2020)
- [Reactive Human-to-Robot Handovers of Arbitrary Objects](https://mlsanity.com/paper/2011.08961/) (2020)
- [In Perfect Shape: Certifiably Optimal 3D Shape Reconstruction from 2D Landmarks](https://mlsanity.com/paper/1911.11924/) (2019)
- [Object-Independent Human-to-Robot Handovers using Real Time Robotic Vision](https://mlsanity.com/paper/2006.01797/) (2020)
- [See the Glass Half Full: Reasoning about Liquid Containers, their Volume and Content](https://mlsanity.com/paper/1701.02718/) (2017)
- [Making Sense of Audio Vibration for Liquid Height Estimation in Robotic Pouring](https://mlsanity.com/paper/1903.00650/) (2019)
- [Accurate Pouring with an Autonomous Robot Using an RGB-D Camera](https://mlsanity.com/paper/1810.03303/) (2018)
- [Reasoning About Liquids via Closed-Loop Simulation](https://mlsanity.com/paper/1703.01656/) (2017)
- [Multi-view shape estimation of transparent containers](https://mlsanity.com/paper/1911.12354/) (2019)
- [Robust Robotic Pouring using Audition and Haptics](https://mlsanity.com/paper/2003.00342/) (2020)
- [Improving filling level classification with adversarial training](https://mlsanity.com/paper/2102.04057/) (2021)
- [Top-1 CORSMAL Challenge 2020 Submission: Filling Mass Estimation Using Multi-modal Observations of Human-robot Handovers](https://mlsanity.com/paper/2012.01311/) (2020)

## Built on by (most cited first, 6 of 6)

- [A mixed-reality dataset for category-level 6D pose and size estimation of hand-occluded containers](https://mlsanity.com/paper/2211.10470/) (2022)
- [Segmenting Object Affordances: Reproducibility and Sensitivity to Scale](https://mlsanity.com/paper/2409.01814/) (2024)
- [Container Localisation and Mass Estimation with an RGB-D Camera](https://mlsanity.com/paper/2203.01207/) (2022)
- [Audio-Visual Object Classification for Human-Robot Collaboration](https://mlsanity.com/paper/2203.01977/) (2022)
- [Stereo Hand-Object Reconstruction for Human-to-Robot Handover](https://mlsanity.com/paper/2412.07487/) (2024)
- [Visual Affordance Prediction: Survey and Reproducibility](https://mlsanity.com/paper/2505.05074/) (2025)

## Abstract (arXiv)

The contactless estimation of the weight of a container and the amount of its content manipulated by a person are key pre-requisites for safe human-to-robot handovers. However, opaqueness and transparencies of the container and the content, and variability of materials, shapes, and sizes, make this estimation difficult. In this paper, we present a range of methods and an open framework to benchmark acoustic and visual perception for the estimation of the capacity of a container, and the type, mass, and amount of its content. The framework includes a dataset, specific tasks and performance measures. We conduct an in-depth comparative analysis of methods that used this framework and audio-only or vision-only baselines designed from related works. Based on this analysis, we can conclude that audio-only and audio-visual classifiers are suitable for the estimation of the type and amount of the content using different types of convolutional neural networks, combined with either recurrent neural networks or a majority voting strategy, whereas computer vision methods are suitable to determine the capacity of the container using regression and geometric approaches. Classifying the content type and level using only audio achieves a weighted average F1-score up to 81% and 97%, respectively. Estimating the container capacity with vision-only approaches and estimating the filling mass with audio-visual multi-stage approaches reach up to 65% weighted average capacity and mass scores. These results show that there is still room for improvement on the design of new methods. These new methods can be ranked and compared on the individual leaderboards provided by our open framework.