# Audio-Visual Object Classification for Human-Robot Collaboration

arXiv: https://arxiv.org/abs/2203.01977 · HTML: https://mlsanity.com/paper/2203.01977/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 27 references that are in the map.
- First posted on arXiv 3 Mar 2022.

## Builds on (most cited first, 15 of 15)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Object Handovers: a Review for Robotics](https://mlsanity.com/paper/2007.12952/) (2020)
- [Reactive Human-to-Robot Handovers of Arbitrary Objects](https://mlsanity.com/paper/2011.08961/) (2020)
- [Object-Independent Human-to-Robot Handovers using Real Time Robotic Vision](https://mlsanity.com/paper/2006.01797/) (2020)
- [See the Glass Half Full: Reasoning about Liquid Containers, their Volume and Content](https://mlsanity.com/paper/1701.02718/) (2017)
- [Towards safe human-to-robot handovers of unknown containers](https://mlsanity.com/paper/2107.01309/) (2021)
- [Multi-view shape estimation of transparent containers](https://mlsanity.com/paper/1911.12354/) (2019)
- [Robust Robotic Pouring using Audition and Haptics](https://mlsanity.com/paper/2003.00342/) (2020)
- [The CORSMAL benchmark for the prediction of the properties of containers](https://mlsanity.com/paper/2107.12719/) (2021)
- [Improving filling level classification with adversarial training](https://mlsanity.com/paper/2102.04057/) (2021)
- [Top-1 CORSMAL Challenge 2020 Submission: Filling Mass Estimation Using Multi-modal Observations of Human-robot Handovers](https://mlsanity.com/paper/2012.01311/) (2020)
- [Audio classification of the content of food containers and drinking glasses](https://mlsanity.com/paper/2103.15999/) (2021)
- [Improving Generalization of Deep Networks for Estimating Physical Properties of Containers and Fillings](https://mlsanity.com/paper/2203.01192/) (2022)
- [Container Localisation and Mass Estimation with an RGB-D Camera](https://mlsanity.com/paper/2203.01207/) (2022)

## Built on by (most cited first, 2 of 2)

- [A mixed-reality dataset for category-level 6D pose and size estimation of hand-occluded containers](https://mlsanity.com/paper/2211.10470/) (2022)
- [Visual Affordance Prediction: Survey and Reproducibility](https://mlsanity.com/paper/2505.05074/) (2025)

## Abstract (arXiv)

Human-robot collaboration requires the contactless estimation of the physical properties of containers manipulated by a person, for example while pouring content in a cup or moving a food box. Acoustic and visual signals can be used to estimate the physical properties of such objects, which may vary substantially in shape, material and size, and also be occluded by the hands of the person. To facilitate comparisons and stimulate progress in solving this problem, we present the CORSMAL challenge and a dataset to assess the performance of the algorithms through a set of well-defined performance scores. The tasks of the challenge are the estimation of the mass, capacity, and dimensions of the object (container), and the classification of the type and amount of its content. A novel feature of the challenge is our real-to-simulation framework for visualising and assessing the impact of estimation errors in human-to-robot handovers.