# ViHOPE: Visuotactile In-Hand Object 6D Pose Estimation with Shape Completion

arXiv: https://arxiv.org/abs/2309.05662 · HTML: https://mlsanity.com/paper/2309.05662/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 40 references that are in the map.
- First posted on arXiv 11 Sep 2023.

## Builds on (most cited first, 29 of 29)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [Least Squares Generative Adversarial Networks](https://mlsanity.com/paper/1611.04076/) (2016)
- [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017)
- [Open3D: A Modern Library for 3D Data Processing](https://mlsanity.com/paper/1801.09847/) (2018)
- [Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation](https://mlsanity.com/paper/1901.02970/) (2019)
- [DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusion](https://mlsanity.com/paper/1901.04780/) (2019)
- [DeepIM: Deep Iterative Matching for 6D Pose Estimation](https://mlsanity.com/paper/1804.00175/) (2018)
- [Pix2Pose: Pixel-Wise Coordinate Regression of Objects for 6D Pose Estimation](https://mlsanity.com/paper/1908.07433/) (2019)
- [Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputs](https://mlsanity.com/paper/1703.09438/) (2017)
- [A System for General In-Hand Object Re-Orientation](https://mlsanity.com/paper/2111.03043/) (2021)
- [Socially Aware Motion Planning with Deep Reinforcement Learning](https://mlsanity.com/paper/1703.08862/) (2017)
- [Synergies Between Affordance and Geometry: 6-DoF Grasp Detection via Implicit Representations](https://mlsanity.com/paper/2104.01542/) (2021)
- [Navigating to Objects in the Real World](https://mlsanity.com/paper/2212.00922/) (2022)
- [Segmentation-driven 6D Object Pose Estimation](https://mlsanity.com/paper/1812.02541/) (2018)
- [From One Hand to Multiple Hands: Imitation Learning for Dexterous Manipulation from Single-Camera Teleoperation](https://mlsanity.com/paper/2204.12490/) (2022)
- [Shape Completion Enabled Robotic Grasping](https://mlsanity.com/paper/1609.08546/) (2016)
- [EPro-PnP: Generalized End-to-End Probabilistic Perspective-n-Points for Monocular Object Pose Estimation](https://mlsanity.com/paper/2303.12787/) (2023)
- [Unsupervised 3D Shape Completion through GAN Inversion](https://mlsanity.com/paper/2104.13366/) (2021)
- [Multimodal Shape Completion via Conditional Generative Adversarial Networks](https://mlsanity.com/paper/2003.07717/) (2020)
- [SSCNav: Confidence-Aware Semantic Scene Completion for Visual Semantic Navigation](https://mlsanity.com/paper/2012.04512/) (2020)
- [6D Object Pose Regression via Supervised Learning on Point Clouds](https://mlsanity.com/paper/2001.08942/) (2020)
- [3D Shape Perception from Monocular Vision, Touch, and Shape Priors](https://mlsanity.com/paper/1808.03247/) (2018)
- [Self-Supervised Geometric Correspondence for Category-Level 6D Object Pose Estimation in the Wild](https://mlsanity.com/paper/2210.07199/) (2022)
- [Tactile Object Pose Estimation from the First Touch with Geometric Contact Rendering](https://mlsanity.com/paper/2012.05205/) (2020)
- [Learning a Structured Latent Space for Unsupervised Point Cloud Completion](https://mlsanity.com/paper/2203.15580/) (2022)
- [Multi-Modal Geometric Learning for Grasping and Manipulation](https://mlsanity.com/paper/1803.07671/) (2018)
- [StereoVoxelNet: Real-Time Obstacle Detection Based on Occupancy Voxels from a Stereo Camera Using Deep Neural Networks](https://mlsanity.com/paper/2209.08459/) (2022)

## Built on by (most cited first, 14 of 14)

- [Binding Touch to Everything: Learning Unified Multimodal Tactile Representations](https://mlsanity.com/paper/2401.18084/) (2024)
- [Robo-ABC: Affordance Generalization Beyond Categories via Semantic Correspondence for Robot Manipulation](https://mlsanity.com/paper/2401.07487/) (2024)
- [Tactile-Augmented Radiance Fields](https://mlsanity.com/paper/2405.04534/) (2024)
- [6DOPE-GS: Online 6D Object Pose Estimation using Gaussian Splatting](https://mlsanity.com/paper/2412.01543/) (2024)
- [FoundHand: Large-Scale Domain-Specific Learning for Controllable Hand Image Generation](https://mlsanity.com/paper/2412.02690/) (2024)
- [HyperTaxel: Hyper-Resolution for Taxel-Based Tactile Signals Through Contrastive Learning](https://mlsanity.com/paper/2408.08312/) (2024)
- [V-HOP: Visuo-Haptic 6D Object Pose Tracking](https://mlsanity.com/paper/2502.17434/) (2025)
- [ViTa-Zero: Zero-shot Visuotactile Object 6D Pose Estimation](https://mlsanity.com/paper/2504.13179/) (2025)
- [ViTaSCOPE: Visuo-tactile Implicit Representation for In-hand Pose and Extrinsic Contact Estimation](https://mlsanity.com/paper/2506.12239/) (2025)
- [Blind Dexterous Grasping via Real2Sim2Real Tactile Policy Learning](https://mlsanity.com/paper/2606.11767/) (2026)
- [RA-Touch: Retrieval-Augmented Touch Understanding with Enriched Visual Data](https://mlsanity.com/paper/2505.14270/) (2025)
- [RETRO: REthinking Tactile Representation Learning with Material PriOrs](https://mlsanity.com/paper/2505.14319/) (2025)
- [UniTac2Pose: A Unified Approach Learned in Simulation for Category-level Visuotactile In-hand Pose Estimation](https://mlsanity.com/paper/2509.15934/) (2025)
- [Tac2Pix: Image-Space Visuo-Tactile Fusion for Dexterous Manipulation](https://mlsanity.com/paper/2606.08765/) (2026)

## Abstract (arXiv)

In this letter, we introduce ViHOPE, a novel framework for estimating the 6D pose of an in-hand object using visuotactile perception. Our key insight is that the accuracy of the 6D object pose estimate can be improved by explicitly completing the shape of the object. To this end, we introduce a novel visuotactile shape completion module that uses a conditional Generative Adversarial Network to complete the shape of an in-hand object based on volumetric representation. This approach improves over prior works that directly regress visuotactile observations to a 6D pose. By explicitly completing the shape of the in-hand object and jointly optimizing the shape completion and pose estimation tasks, we improve the accuracy of the 6D object pose estimate. We train and test our model on a synthetic dataset and compare it with the state-of-the-art. In the visuotactile shape completion task, we outperform the state-of-the-art by 265% using the Intersection of Union metric and achieve 88% lower Chamfer Distance. In the visuotactile pose estimation task, we present results that suggest our framework reduces position and angular errors by 35% and 64%, respectively. Furthermore, we ablate our framework to confirm the gain on the 6D object pose estimate from explicitly completing the shape. Ultimately, we show that our framework produces models that are robust to sim-to-real transfer on a real-world robot platform.