# Generating Future Observations to Estimate Grasp Success in Cluttered Environments

arXiv: https://arxiv.org/abs/2403.07877 · HTML: https://mlsanity.com/paper/2403.07877/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 7 references that are in the map.
- First posted on arXiv 18 Dec 2023.

## Builds on (most cited first, 7 of 7)

- [Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection](https://mlsanity.com/paper/1603.02199/) (2016)
- [Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control](https://mlsanity.com/paper/1812.00568/) (2018)
- [MaskViT: Masked Visual Pre-Training for Video Prediction](https://mlsanity.com/paper/2206.11894/) (2022)
- [GelTip: A Finger-shaped Optical Tactile Sensor for Robotic Manipulation](https://mlsanity.com/paper/2008.05404/) (2020)
- [Generation of GelSight Tactile Images for Sim2Real Learning](https://mlsanity.com/paper/2101.07169/) (2021)
- [Beyond Flat GelSight Sensors: Simulation of Optical Tactile Sensors of Complex Morphologies for Sim2Real Learning](https://mlsanity.com/paper/2305.12605/) (2023)
- [GarmNet: Improving Global with Local Perception for Robotic Laundry Folding](https://mlsanity.com/paper/1907.00408/) (2019)

## Abstract (arXiv)

End-to-end self-supervised models have been proposed for estimating the success of future candidate grasps and video predictive models for generating future observations. However, none have yet studied these two strategies side-by-side for addressing the aforementioned grasping problem. We investigate and compare a model-free approach, to estimate the success of a candidate grasp, against a model-based alternative that exploits a self-supervised learnt predictive model that generates a future observation of the gripper about to grasp an object. Our experiments demonstrate that despite the end-to-end model-free model obtaining a best accuracy of 72%, the proposed model-based pipeline yields a significantly higher accuracy of 82%.