# Simultaneous Multi-View Object Recognition and Grasping in Open-Ended Domains

arXiv: https://arxiv.org/abs/2106.01866 · HTML: https://mlsanity.com/paper/2106.01866/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 64 references that are in the map.
- First posted on arXiv 3 Jun 2021.

## Builds on (most cited first, 25 of 25)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [YOLOv4: Optimal Speed and Accuracy of Object Detection](https://mlsanity.com/paper/2004.10934/) (2020)
- [Deep Bayesian Active Learning with Image Data](https://mlsanity.com/paper/1703.02910/) (2017)
- [Multi-view Convolutional Neural Networks for 3D Shape Recognition](https://mlsanity.com/paper/1505.00880/) (2015)
- [A Survey of Deep Active Learning](https://mlsanity.com/paper/2009.00236/) (2020)
- [Towards Open Set Deep Networks](https://mlsanity.com/paper/1511.06233/) (2015)
- [Soft Rasterizer: A Differentiable Renderer for Image-based 3D Reasoning](https://mlsanity.com/paper/1904.01786/) (2019)
- [Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics](https://mlsanity.com/paper/1703.09312/) (2017)
- [6-DOF GraspNet: Variational Grasp Generation for Object Manipulation](https://mlsanity.com/paper/1905.10520/) (2019)
- [Volumetric and Multi-View CNNs for Object Classification on 3D Data](https://mlsanity.com/paper/1604.03265/) (2016)
- [Escape from Cells: Deep Kd-Networks for the Recognition of 3D Point Cloud Models](https://mlsanity.com/paper/1704.01222/) (2017)
- [Context Prior for Scene Segmentation](https://mlsanity.com/paper/2004.01547/) (2020)
- [Closing the Loop for Robotic Grasping: A Real-time, Generative Grasp Synthesis Approach](https://mlsanity.com/paper/1804.05172/) (2018)
- [Deep Learning for Detecting Robotic Grasps](https://mlsanity.com/paper/1301.3592/) (2013)
- [Grasp Pose Detection in Point Clouds](https://mlsanity.com/paper/1706.09911/) (2017)
- [Data-Driven Grasp Synthesis - A Survey](https://mlsanity.com/paper/1309.2660/) (2013)
- [ViewAL: Active Learning with Viewpoint Entropy for Semantic Segmentation](https://mlsanity.com/paper/1911.11789/) (2019)
- [Volumetric Grasping Network: Real-time 6 DOF Grasp Detection in Clutter](https://mlsanity.com/paper/2101.01132/) (2021)
- [Antipodal Robotic Grasping using Generative Residual Convolutional Neural Network](https://mlsanity.com/paper/1909.04810/) (2019)
- [Attention Convolutional Binary Neural Tree for Fine-Grained Visual Categorization](https://mlsanity.com/paper/1909.11378/) (2019)
- [High precision grasp pose detection in dense clutter](https://mlsanity.com/paper/1603.01564/) (2016)
- [Multi-View Picking: Next-best-view Reaching for Improved Grasping in Clutter](https://mlsanity.com/paper/1809.08564/) (2018)
- [MORE: Simultaneous Multi-View 3D Object Recognition and Pose Estimation](https://mlsanity.com/paper/2103.09863/) (2021)
- [The State of Lifelong Learning in Service Robots: Current Bottlenecks in Object Perception and Manipulation](https://mlsanity.com/paper/2003.08151/) (2020)

## Built on by (most cited first, 10 of 10)

- [Lifelong Ensemble Learning based on Multiple Representations for Few-Shot Object Recognition](https://mlsanity.com/paper/2205.01982/) (2022)
- [Distracted Robot: How Visual Clutter Undermine Robotic Manipulation](https://mlsanity.com/paper/2511.22780/) (2025)
- [MVSA-Net: Multi-View State-Action Recognition for Robust and Deployable Trajectory Generation](https://mlsanity.com/paper/2311.08393/) (2023)
- [Improving Robotic Manipulation Robustness via NICE Scene Surgery](https://mlsanity.com/paper/2511.22777/) (2025)
- [Lifelong 3D Object Recognition and Grasp Synthesis Using Dual Memory Recurrent Self-Organization Networks](https://mlsanity.com/paper/2109.11544/) (2021)
- [GraspCaps: A Capsule Network Approach for Familiar 6DoF Object Grasping](https://mlsanity.com/paper/2210.03628/) (2022)
- [VITAL: Interactive Few-Shot Imitation Learning via Visual Human-in-the-Loop Corrections](https://mlsanity.com/paper/2407.21244/) (2024)
- [LM-MCVT: A Lightweight Multi-modal Multi-view Convolutional-Vision Transformer Approach for 3D Object Recognition](https://mlsanity.com/paper/2504.19256/) (2025)
- [Optimizing Grasping in Legged Robots: A Deep Learning Approach to Loco-Manipulation](https://mlsanity.com/paper/2508.17466/) (2025)
- [Learning to Throw Objects Safely in Multi-Obstacle Environments](https://mlsanity.com/paper/2607.06388/) (2026)

## Abstract (arXiv)

To aid humans in everyday tasks, robots need to know which objects exist in the scene, where they are, and how to grasp and manipulate them in different situations. Therefore, object recognition and grasping are two key functionalities for autonomous robots. Most state-of-the-art approaches treat object recognition and grasping as two separate problems, even though both use visual input. Furthermore, the knowledge of the robot is fixed after the training phase. In such cases, if the robot encounters new object categories, it must be retrained to incorporate new information without catastrophic forgetting. In order to resolve this problem, we propose a deep learning architecture with an augmented memory capacity to handle open-ended object recognition and grasping simultaneously. In particular, our approach takes multi-views of an object as input and jointly estimates pixel-wise grasp configuration as well as a deep scale- and rotation-invariant representation as output. The obtained representation is then used for open-ended object recognition through a meta-active learning technique. We demonstrate the ability of our approach to grasp never-seen-before objects and to rapidly learn new object categories using very few examples on-site in both simulation and real-world settings. A video of these experiments is available online at: https://youtu.be/n9SMpuEkOgk