# Cluttered Food Grasping with Adaptive Fingers and Synthetic-Data Trained Object Detection

arXiv: https://arxiv.org/abs/2203.05187 · HTML: https://mlsanity.com/paper/2203.05187/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 50 references that are in the map.
- First posted on arXiv 10 Mar 2022.

## Builds on (most cited first, 20 of 20)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomization](https://mlsanity.com/paper/1804.06516/) (2018)
- [Differentiable Rendering: A Survey](https://mlsanity.com/paper/2006.12057/) (2020)
- [Deep Learning for Detecting Robotic Grasps](https://mlsanity.com/paper/1301.3592/) (2013)
- [Grasp Pose Detection in Point Clouds](https://mlsanity.com/paper/1706.09911/) (2017)
- [6-DOF Grasping for Target-driven Object Manipulation in Clutter](https://mlsanity.com/paper/1912.03628/) (2019)
- [A Survey on Food Computing](https://mlsanity.com/paper/1808.07202/) (2018)
- [A Large-Scale Benchmark for Food Image Segmentation](https://mlsanity.com/paper/2105.05409/) (2021)
- [DDGC: Generative Deep Dexterous Grasping in Clutter](https://mlsanity.com/paper/2103.04783/) (2021)
- [Large-scale 6D Object Pose Estimation Dataset for Industrial Bin-Picking](https://mlsanity.com/paper/1912.12125/) (2019)
- [Simultaneous Semantic and Collision Learning for 6-DoF Grasp Pose Estimation](https://mlsanity.com/paper/2108.02425/) (2021)
- [Hierarchical Policies for Cluttered-Scene Grasping with Latent Plans](https://mlsanity.com/paper/2107.01518/) (2021)
- [Collision-Aware Target-Driven Object Grasping in Constrained Environments](https://mlsanity.com/paper/2104.00776/) (2021)
- [MUSEFood: Multi-sensor-based Food Volume Estimation on Smartphones](https://mlsanity.com/paper/1903.07437/) (2019)
- [Uncertainty-Aware Self-Supervised Target-Mass Grasping of Granular Foods](https://mlsanity.com/paper/2105.12946/) (2021)
- [Transferring Experience from Simulation to the Real World for Precise Pick-And-Place Tasks in Highly Cluttered Scenes](https://mlsanity.com/paper/2101.04781/) (2021)
- [Deep Learning based Food Instance Segmentation using Synthetic Data](https://mlsanity.com/paper/2107.07191/) (2021)
- [Efficient Grasp Planning and Execution with Multi-Fingered Hands by Surface Fitting](https://mlsanity.com/paper/1902.10841/) (2019)
- [Target-mass Grasping of Entangled Food using Pre-grasping & Post-grasping](https://mlsanity.com/paper/2201.00933/) (2022)

## Built on by (most cited first, 8 of 8)

- [A survey of synthetic data augmentation methods in computer vision](https://mlsanity.com/paper/2403.10075/) (2024)
- [FoodSky: A Food-oriented Large Language Model that Passes the Chef and Dietetic Examination](https://mlsanity.com/paper/2406.10261/) (2024)
- [SeeDS: Semantic Separable Diffusion Synthesizer for Zero-shot Food Detection](https://mlsanity.com/paper/2310.04689/) (2023)
- [Close the Sim2real Gap via Physically-based Structured Light Synthetic Data Simulation](https://mlsanity.com/paper/2407.12449/) (2024)
- [Distracted Robot: How Visual Clutter Undermine Robotic Manipulation](https://mlsanity.com/paper/2511.22780/) (2025)
- [Improving Robotic Manipulation Robustness via NICE Scene Surgery](https://mlsanity.com/paper/2511.22777/) (2025)
- [Experimental Study on Automatically Assembling Custom Catering Packages With a 3-DOF Delta Robot Using Deep Learning Methods](https://mlsanity.com/paper/2505.11879/) (2025)
- [Synthetic Data Augmentation for Enhanced Chicken Carcass Instance Segmentation](https://mlsanity.com/paper/2507.18558/) (2025)

## Abstract (arXiv)

The food packaging industry handles an immense variety of food products with wide-ranging shapes and sizes, even within one kind of food. Menus are also diverse and change frequently, making automation of pick-and-place difficult. A popular approach to bin-picking is to first identify each piece of food in the tray by using an instance segmentation method. However, human annotations to train these methods are unreliable and error-prone since foods are packed close together with unclear boundaries and visual similarity making separation of pieces difficult. To address this problem, we propose a method that trains purely on synthetic data and successfully transfers to the real world using sim2real methods by creating datasets of filled food trays using high-quality 3d models of real pieces of food for the training instance segmentation models. Another concern is that foods are easily damaged during grasping. We address this by introducing two additional methods -- a novel adaptive finger mechanism to passively retract when a collision occurs, and a method to filter grasps that are likely to cause damage to neighbouring pieces of food during a grasp. We demonstrate the effectiveness of the proposed method on several kinds of real foods.