# Uncertainty-driven Exploration Strategies for Online Grasp Learning

arXiv: https://arxiv.org/abs/2309.12038 · HTML: https://mlsanity.com/paper/2309.12038/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 58 references that are in the map.
- First posted on arXiv 21 Sep 2023.

## Builds on (most cited first, 30 of 38)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation](https://mlsanity.com/paper/1806.10293/) (2018)
- [Distributional Reinforcement Learning with Quantile Regression](https://mlsanity.com/paper/1710.10044/) (2017)
- [Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection](https://mlsanity.com/paper/1603.02199/) (2016)
- [Distributed Prioritized Experience Replay](https://mlsanity.com/paper/1803.00933/) (2018)
- [Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenes](https://mlsanity.com/paper/2103.14127/) (2021)
- [Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble](https://mlsanity.com/paper/2107.00591/) (2021)
- [SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learning](https://mlsanity.com/paper/2007.04938/) (2020)
- [Learning Synergies between Pushing and Grasping with Self-supervised Deep Reinforcement Learning](https://mlsanity.com/paper/1803.09956/) (2018)
- [Robotic Pick-and-Place of Novel Objects in Clutter with Multi-Affordance Grasping and Cross-Domain Image Matching](https://mlsanity.com/paper/1710.01330/) (2017)
- [Closing the Loop for Robotic Grasping: A Real-time, Generative Grasp Synthesis Approach](https://mlsanity.com/paper/1804.05172/) (2018)
- [Synergies Between Affordance and Geometry: 6-DoF Grasp Detection via Implicit Representations](https://mlsanity.com/paper/2104.01542/) (2021)
- [Data-Driven Grasp Synthesis - A Survey](https://mlsanity.com/paper/1309.2660/) (2013)
- [Volumetric Grasping Network: Real-time 6 DOF Grasp Detection in Clutter](https://mlsanity.com/paper/2101.01132/) (2021)
- [Deep Learning Approaches to Grasp Synthesis: A Review](https://mlsanity.com/paper/2207.02556/) (2022)
- [Estimating Risk and Uncertainty in Deep Reinforcement Learning](https://mlsanity.com/paper/1905.09638/) (2019)
- [S4G: Amodal Single-view Single-Shot SE(3) Grasp Detection in Cluttered Scenes](https://mlsanity.com/paper/1910.14218/) (2019)
- [PointNet++ Grasping: Learning An End-to-end Spatial Grasp Generation Algorithm from Sparse Point Clouds](https://mlsanity.com/paper/2003.09644/) (2020)
- [SuctionNet-1Billion: A Large-Scale Benchmark for Suction Grasping](https://mlsanity.com/paper/2103.12311/) (2021)
- [HACMan: Learning Hybrid Actor-Critic Maps for 6D Non-Prehensile Manipulation](https://mlsanity.com/paper/2305.03942/) (2023)
- [Disentangling Epistemic and Aleatoric Uncertainty in Reinforcement Learning](https://mlsanity.com/paper/2206.01558/) (2022)
- [Robot Learning of Shifting Objects for Grasping in Cluttered Environments](https://mlsanity.com/paper/1907.11035/) (2019)
- [Multi-Fingered Active Grasp Learning](https://mlsanity.com/paper/2006.05264/) (2020)
- [MetaGraspNet: A Large-Scale Benchmark Dataset for Scene-Aware Ambidextrous Bin Picking via Physics-based Metaverse Synthesis](https://mlsanity.com/paper/2208.03963/) (2022)
- [Robotic Grasping through Combined Image-Based Grasp Proposal and 3D Reconstruction](https://mlsanity.com/paper/2003.01649/) (2020)
- [Exploratory Grasping: Asymptotically Optimal Algorithms for Grasping Challenging Polyhedral Objects](https://mlsanity.com/paper/2011.05632/) (2020)
- [Kalman Bayesian Neural Networks for Closed-form Online Learning](https://mlsanity.com/paper/2110.00944/) (2021)
- [GDN: A Coarse-To-Fine (C2F) Representation for End-To-End 6-DoF Grasp Detection](https://mlsanity.com/paper/2010.10695/) (2020)

## Built on by (most cited first, 6 of 6)

- [FlowCorrect: Efficient Interactive Correction of Generative Flow Policies for Robotic Manipulation](https://mlsanity.com/paper/2602.22056/) (2026)
- [FFHFlow: Diverse and Uncertainty-Aware Dexterous Grasp Generation via Flow Variational Inference](https://mlsanity.com/paper/2407.15161/) (2024)
- [Pseudo-Labeling and Contextual Curriculum Learning for Online Grasp Learning in Robotic Bin Picking](https://mlsanity.com/paper/2403.02495/) (2024)
- [LaMOuR: Leveraging Language Models for Out-of-Distribution Recovery in Reinforcement Learning](https://mlsanity.com/paper/2503.17125/) (2025)
- [DF-ExpEnse: Diffusion Filtered Exploration for Sample Efficient Finetuning](https://mlsanity.com/paper/2606.19656/) (2026)
- [Continual Learning for 6-DoF Grasp Synthesis via Experience and Demonstrations](https://mlsanity.com/paper/2610.01301/) (2026)

## Abstract (arXiv)

Existing grasp prediction approaches are mostly based on offline learning, while, ignoring the exploratory grasp learning during online adaptation to new picking scenarios, i.e., objects that are unseen or out-of-domain (OOD), camera and bin settings, etc. In this paper, we present an uncertainty-based approach for online learning of grasp predictions for robotic bin picking. Specifically, the online learning algorithm with an effective exploration strategy can significantly improve its adaptation performance to unseen environment settings. To this end, we first propose to formulate online grasp learning as an RL problem that will allow us to adapt both grasp reward prediction and grasp poses. We propose various uncertainty estimation schemes based on Bayesian uncertainty quantification and distributional ensembles. We carry out evaluations on real-world bin picking scenes of varying difficulty. The objects in the bin have various challenging physical and perceptual characteristics that can be characterized by semi- or total transparency, and irregular or curved surfaces. The results of our experiments demonstrate a notable improvement of grasp performance in comparison to conventional online learning methods which incorporate only naive exploration strategies. Video: https://youtu.be/fPKOrjC2QrU