# Collision-aware In-hand 6D Object Pose Estimation using Multiple Vision-based Tactile Sensors

arXiv: https://arxiv.org/abs/2301.13667 · HTML: https://mlsanity.com/paper/2301.13667/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 36 references that are in the map.
- First posted on arXiv 31 Jan 2023.

## Builds on (most cited first, 24 of 24)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017)
- [PVNet: Pixel-wise Voting Network for 6DoF Pose Estimation](https://mlsanity.com/paper/1812.11788/) (2018)
- [Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objects](https://mlsanity.com/paper/1809.10790/) (2018)
- [Implicit 3D Orientation Learning for 6D Object Detection from RGB Images](https://mlsanity.com/paper/1902.01275/) (2019)
- [DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation](https://mlsanity.com/paper/2005.14679/) (2020)
- [HybridPose: 6D Object Pose Estimation under Hybrid Representations](https://mlsanity.com/paper/2001.01869/) (2020)
- [PoseRBPF: A Rao-Blackwellized Particle Filter for 6D Object Pose Tracking](https://mlsanity.com/paper/1905.09304/) (2019)
- [se(3)-TrackNet: Data-driven 6D Pose Tracking by Calibrating Image Residuals in Synthetic Domains](https://mlsanity.com/paper/2007.13866/) (2020)
- [ObjectFolder 2.0: A Multisensory Object Dataset for Sim2Real Transfer](https://mlsanity.com/paper/2204.02389/) (2022)
- [TACTO: A Fast, Flexible, and Open-source Simulator for High-Resolution Vision-based Tactile Sensors](https://mlsanity.com/paper/2012.08456/) (2020)
- [Taxim: An Example-based Simulation Model for GelSight Tactile Sensors](https://mlsanity.com/paper/2109.04027/) (2021)
- [Single-Stage Keypoint-Based Category-Level Object Pose Estimation from an RGB Image](https://mlsanity.com/paper/2109.06161/) (2021)
- [Iterative Corresponding Geometry: Fusing Region and Depth for Highly Efficient 3D Tracking of Textureless Objects](https://mlsanity.com/paper/2203.05334/) (2022)
- [Tactile Object Pose Estimation from the First Touch with Geometric Contact Rendering](https://mlsanity.com/paper/2012.05205/) (2020)
- [Keypoint-Based Category-Level Object Pose Tracking from an RGB Sequence with Uncertainty Estimation](https://mlsanity.com/paper/2205.11047/) (2022)
- [Tactile SLAM: Real-time inference of shape and pose from planar pushing](https://mlsanity.com/paper/2011.07044/) (2020)
- [In-Hand Object Pose Tracking via Contact Feedback and GPU-Accelerated Robotic Simulation](https://mlsanity.com/paper/2002.12160/) (2020)
- [ROFT: Real-Time Optical Flow-Aided 6D Object Pose and Velocity Tracking](https://mlsanity.com/paper/2111.03821/) (2021)
- [PatchGraph: In-hand tactile tracking with learned surface normals](https://mlsanity.com/paper/2111.07524/) (2021)
- [Tactile Pose Estimation and Policy Learning for Unknown Object Manipulation](https://mlsanity.com/paper/2203.10685/) (2022)
- [Active Visuo-Tactile Interactive Robotic Perception for Accurate Object Pose Estimation in Dense Clutter](https://mlsanity.com/paper/2202.02207/) (2022)
- [Simultaneous Contact Location and Object Pose Estimation Using Proprioception and Tactile Feedback](https://mlsanity.com/paper/2206.01245/) (2022)

## Built on by (most cited first, 10 of 10)

- [Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AI](https://mlsanity.com/paper/2407.06886/) (2024)
- [AcTExplore: Active Tactile Exploration of Unknown Objects](https://mlsanity.com/paper/2310.08745/) (2023)
- [FeelAnyForce: Estimating Contact Force Feedback from Tactile Sensation for Vision-Based Tactile Sensors](https://mlsanity.com/paper/2410.02048/) (2024)
- [HyperTaxel: Hyper-Resolution for Taxel-Based Tactile Signals Through Contrastive Learning](https://mlsanity.com/paper/2408.08312/) (2024)
- [Tactile-Filter: Interactive Tactile Perception for Part Mating](https://mlsanity.com/paper/2303.06034/) (2023)
- [Sim2Real Bilevel Adaptation for Object Surface Classification using Vision-Based Tactile Sensors](https://mlsanity.com/paper/2311.01380/) (2023)
- [ViTa-Zero: Zero-shot Visuotactile Object 6D Pose Estimation](https://mlsanity.com/paper/2504.13179/) (2025)
- [Visuo-Tactile Object Pose Estimation for a Multi-Finger Robot Hand with Low-Resolution In-Hand Tactile Sensing](https://mlsanity.com/paper/2503.19893/) (2025)
- [Object Pose Estimation through Dexterous Touch](https://mlsanity.com/paper/2509.13591/) (2025)
- [Physically Grounded 3D Generative Reconstruction under Hand Occlusion using Proprioception and Multi-Contact Touch](https://mlsanity.com/paper/2604.09100/) (2026)

## Abstract (arXiv)

In this paper, we address the problem of estimating the in-hand 6D pose of an object in contact with multiple vision-based tactile sensors. We reason on the possible spatial configurations of the sensors along the object surface. Specifically, we filter contact hypotheses using geometric reasoning and a Convolutional Neural Network (CNN), trained on simulated object-agnostic images, to promote those that better comply with the actual tactile images from the sensors. We use the selected sensors configurations to optimize over the space of 6D poses using a Gradient Descent-based approach. We finally rank the obtained poses by penalizing those that are in collision with the sensors. We carry out experiments in simulation using the DIGIT vision-based sensor with several objects, from the standard YCB model set. The results demonstrate that our approach estimates object poses that are compatible with actual object-sensor contacts in 87.5% of cases while reaching an average positional error in the order of 2 centimeters. Our analysis also includes qualitative results of experiments with a real DIGIT sensor.