# Enhancing Generalizable 6D Pose Tracking of an In-Hand Object with Tactile Sensing

arXiv: https://arxiv.org/abs/2210.04026 · HTML: https://mlsanity.com/paper/2210.04026/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 55 references that are in the map.
- First posted on arXiv 8 Oct 2022.

## Builds on (most cited first, 30 of 34)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017)
- [SAPIEN: A SimulAted Part-based Interactive ENvironment](https://mlsanity.com/paper/2003.08515/) (2020)
- [Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation](https://mlsanity.com/paper/1901.02970/) (2019)
- [PoinTr: Diverse Point Cloud Completion with Geometry-Aware Transformers](https://mlsanity.com/paper/2108.08839/) (2021)
- [PVN3D: A Deep Point-wise 3D Keypoints Voting Network for 6DoF Pose Estimation](https://mlsanity.com/paper/1911.04231/) (2019)
- [DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation](https://mlsanity.com/paper/2005.14679/) (2020)
- [Modular Interactive Video Object Segmentation: Interaction-to-Mask, Propagation and Difference-Aware Fusion](https://mlsanity.com/paper/2103.07941/) (2021)
- [PoseRBPF: A Rao-Blackwellized Particle Filter for 6D Object Pose Tracking](https://mlsanity.com/paper/1905.09304/) (2019)
- [se(3)-TrackNet: Data-driven 6D Pose Tracking by Calibrating Image Residuals in Synthetic Domains](https://mlsanity.com/paper/2007.13866/) (2020)
- [BundleTrack: 6D Pose Tracking for Novel Objects without Instance or Category-Level 3D Models](https://mlsanity.com/paper/2108.00516/) (2021)
- [CAPTRA: CAtegory-level Pose Tracking for Rigid and Articulated Objects from Point Clouds](https://mlsanity.com/paper/2104.03437/) (2021)
- [6-PACK: Category-level 6D Pose Tracker with Anchor-Based Keypoints](https://mlsanity.com/paper/1910.10750/) (2019)
- [ObjectFolder 2.0: A Multisensory Object Dataset for Sim2Real Transfer](https://mlsanity.com/paper/2204.02389/) (2022)
- [ObjectFolder: A Dataset of Objects with Implicit Visual, Auditory, and Tactile Representations](https://mlsanity.com/paper/2109.07991/) (2021)
- [GelSlim: A High-Resolution, Compact, Robust, and Calibrated Tactile-sensing Finger](https://mlsanity.com/paper/1803.00628/) (2018)
- [3D Shape Reconstruction from Vision and Touch](https://mlsanity.com/paper/2007.03778/) (2020)
- [GelSlim3.0: High-Resolution Measurement of Shape, Force and Slip in a Compact Tactile-Sensing Finger](https://mlsanity.com/paper/2103.12269/) (2021)
- [Active 3D Shape Reconstruction from Vision and Touch](https://mlsanity.com/paper/2107.09584/) (2021)
- [3D Shape Perception from Monocular Vision, Touch, and Shape Priors](https://mlsanity.com/paper/1808.03247/) (2018)
- [ShapeMap 3-D: Efficient shape mapping through dense touch and vision](https://mlsanity.com/paper/2109.09884/) (2021)
- [OmniTact: A Multi-Directional High Resolution Touch Sensor](https://mlsanity.com/paper/2003.06965/) (2020)
- [Robust, Occlusion-aware Pose Estimation for Objects Grasped by Adaptive Hands](https://mlsanity.com/paper/2003.03518/) (2020)
- [Reactive Human-to-Robot Handovers of Arbitrary Objects](https://mlsanity.com/paper/2011.08961/) (2020)
- [Keypoint-Based Category-Level Object Pose Tracking from an RGB Sequence with Uncertainty Estimation](https://mlsanity.com/paper/2205.11047/) (2022)
- [Close the Optical Sensing Domain Gap by Physics-Grounded Active Stereo Sensor Simulation](https://mlsanity.com/paper/2201.11924/) (2022)
- [PIZZA: A Powerful Image-only Zero-Shot Zero-CAD Approach to 6 DoF Tracking](https://mlsanity.com/paper/2209.07589/) (2022)
- [It Takes Two: Learning to Plan for Human-Robot Cooperative Carrying](https://mlsanity.com/paper/2209.12890/) (2022)

## Built on by (most cited first, 5 of 5)

- [ManiWAV: Learning Robot Manipulation from In-the-Wild Audio-Visual Data](https://mlsanity.com/paper/2406.19464/) (2024)
- [Compliant Residual DAgger: Improving Real-World Contact-Rich Manipulation with Human Corrections](https://mlsanity.com/paper/2506.16685/) (2025)
- [V-HOP: Visuo-Haptic 6D Object Pose Tracking](https://mlsanity.com/paper/2502.17434/) (2025)
- [ViTa-Zero: Zero-shot Visuotactile Object 6D Pose Estimation](https://mlsanity.com/paper/2504.13179/) (2025)
- [Object Pose Estimation through Dexterous Touch](https://mlsanity.com/paper/2509.13591/) (2025)

## Abstract (arXiv)

When manipulating an object to accomplish complex tasks, humans rely on both vision and touch to keep track of the object's 6D pose. However, most existing object pose tracking systems in robotics rely exclusively on visual signals, which hinder a robot's ability to manipulate objects effectively. To address this limitation, we introduce TEG-Track, a tactile-enhanced 6D pose tracking system that can track previously unseen objects held in hand. From consecutive tactile signals, TEG-Track optimizes object velocities from marker flows when slippage does not occur, or regresses velocities using a slippage estimation network when slippage is detected. The estimated object velocities are integrated into a geometric-kinematic optimization scheme to enhance existing visual pose trackers. To evaluate our method and to facilitate future research, we construct a real-world dataset for visual-tactile in-hand object pose tracking. Experimental results demonstrate that TEG-Track consistently enhances state-of-the-art generalizable 6D pose trackers in synthetic and real-world scenarios. Our code and dataset are available at https://github.com/leolyliu/TEG-Track.