# Tac2Pose: Tactile Object Pose Estimation from the First Touch

arXiv: https://arxiv.org/abs/2204.11701 · HTML: https://mlsanity.com/paper/2204.11701/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 58 references that are in the map.
- First posted on arXiv 25 Apr 2022.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [CosyPose: Consistent multi-view multi-object 6D pose estimation](https://mlsanity.com/paper/2008.08465/) (2020)
- [DeepIM: Deep Iterative Matching for 6D Pose Estimation](https://mlsanity.com/paper/1804.00175/) (2018)
- [Implicit 3D Orientation Learning for 6D Object Detection from RGB Images](https://mlsanity.com/paper/1902.01275/) (2019)
- [Robotic Pick-and-Place of Novel Objects in Clutter with Multi-Affordance Grasping and Cross-Domain Image Matching](https://mlsanity.com/paper/1710.01330/) (2017)
- [PoseRBPF: A Rao-Blackwellized Particle Filter for 6D Object Pose Tracking](https://mlsanity.com/paper/1905.09304/) (2019)
- [FilterReg: Robust and Efficient Probabilistic Point-Set Registration using Gaussian Filter and Twist Parameterization](https://mlsanity.com/paper/1811.10136/) (2018)
- [GelSlim3.0: High-Resolution Measurement of Shape, Force and Slip in a Compact Tactile-Sensing Finger](https://mlsanity.com/paper/2103.12269/) (2021)
- [Manipulation by Feel: Touch-Based Control with Deep Predictive Models](https://mlsanity.com/paper/1903.04128/) (2019)
- [3D Shape Perception from Monocular Vision, Touch, and Shape Priors](https://mlsanity.com/paper/1808.03247/) (2018)
- [Robotic Tactile Perception of Object Properties: A Review](https://mlsanity.com/paper/1711.03810/) (2017)
- [Tactile Object Pose Estimation from the First Touch with Geometric Contact Rendering](https://mlsanity.com/paper/2012.05205/) (2020)
- [Tactile Mapping and Localization from High-Resolution Tactile Imprints](https://mlsanity.com/paper/1904.10944/) (2019)
- [Active Extrinsic Contact Sensing: Application to General Peg-in-Hole Insertion](https://mlsanity.com/paper/2110.03555/) (2021)
- [Tactile Regrasp: Grasp Adjustments via Simulated Tactile Transformations](https://mlsanity.com/paper/1803.01940/) (2018)
- [Fast Object Learning and Dual-arm Coordination for Cluttered Stowing, Picking, and Packing](https://mlsanity.com/paper/1810.02977/) (2018)
- [From pixels to percepts: Highly robust edge perception and contour following using deep learning and an optical biomimetic tactile sensor](https://mlsanity.com/paper/1812.02941/) (2018)
- [Localizing the Object Contact through Matching Tactile Features with Visual Map](https://mlsanity.com/paper/1708.04441/) (2017)
- [PatchGraph: In-hand tactile tracking with learned surface normals](https://mlsanity.com/paper/2111.07524/) (2021)
- [The Manifold Particle Filter for State Estimation on High-dimensional Implicit Manifolds](https://mlsanity.com/paper/1604.07224/) (2016)
- [Semantic Segmentation from Limited Training Data](https://mlsanity.com/paper/1709.07665/) (2017)
- [Realtime State Estimation with Tactile and Visual sensing. Application to Planar Manipulation](https://mlsanity.com/paper/1709.09694/) (2017)
- [PrimA6D: Rotational Primitive Reconstruction for Enhanced and Robust 6D Pose Estimation](https://mlsanity.com/paper/2006.07789/) (2020)
- [Learning Tactile Models for Factor Graph-based Estimation](https://mlsanity.com/paper/2012.03768/) (2020)

## Built on by (most cited first, 15 of 15)

- [Neural Contact Fields: Tracking Extrinsic Contact with Tactile Sensing](https://mlsanity.com/paper/2210.09297/) (2022)
- [FingerSLAM: Closed-loop Unknown Object Localization and Reconstruction from Visuo-tactile Feedback](https://mlsanity.com/paper/2303.07997/) (2023)
- [AcTExplore: Active Tactile Exploration of Unknown Objects](https://mlsanity.com/paper/2310.08745/) (2023)
- [GelSight Svelte: A Human Finger-shaped Single-camera Tactile Robot Finger with Large Sensing Coverage and Proprioceptive Sensing](https://mlsanity.com/paper/2309.10885/) (2023)
- [Taccel: Scaling Up Vision-based Tactile Robotics via High-performance GPU Simulation](https://mlsanity.com/paper/2504.12908/) (2025)
- [GelSLAM: A Real-time, High-Fidelity, and Robust 3D Tactile SLAM System](https://mlsanity.com/paper/2508.15990/) (2025)
- [Tactile-Filter: Interactive Tactile Perception for Part Mating](https://mlsanity.com/paper/2303.06034/) (2023)
- [ViTaSCOPE: Visuo-tactile Implicit Representation for In-hand Pose and Extrinsic Contact Estimation](https://mlsanity.com/paper/2506.12239/) (2025)
- [NLiPsCalib: An Efficient Calibration Framework for High-Fidelity 3D Reconstruction of Curved Visuotactile Sensors](https://mlsanity.com/paper/2603.09319/) (2026)
- [Visuo-Tactile Object Pose Estimation for a Multi-Finger Robot Hand with Low-Resolution In-Hand Tactile Sensing](https://mlsanity.com/paper/2503.19893/) (2025)
- [PseudoTouch: Efficiently Imaging the Surface Feel of Objects for Robotic Manipulation](https://mlsanity.com/paper/2403.15107/) (2024)
- [ControlTac: Scaling Tactile Data with Physically Controlled Tactile Image Generation](https://mlsanity.com/paper/2505.20498/) (2025)
- [UniTac2Pose: A Unified Approach Learned in Simulation for Category-level Visuotactile In-hand Pose Estimation](https://mlsanity.com/paper/2509.15934/) (2025)
- [VTLoc: Learning-based Tactile Contact Localization in Visual Point Clouds](https://mlsanity.com/paper/2607.16146/) (2026)
- [Robot Learning with Visual Predicted Force](https://mlsanity.com/paper/2610.04741/) (2026)

## Abstract (arXiv)

In this paper, we present Tac2Pose, an object-specific approach to tactile pose estimation from the first touch for known objects. Given the object geometry, we learn a tailored perception model in simulation that estimates a probability distribution over possible object poses given a tactile observation. To do so, we simulate the contact shapes that a dense set of object poses would produce on the sensor. Then, given a new contact shape obtained from the sensor, we match it against the pre-computed set using an object-specific embedding learned using contrastive learning. We obtain contact shapes from the sensor with an object-agnostic calibration step that maps RGB tactile observations to binary contact shapes. This mapping, which can be reused across object and sensor instances, is the only step trained with real sensor data. This results in a perception model that localizes objects from the first real tactile observation. Importantly, it produces pose distributions and can incorporate additional pose constraints coming from other perception systems, contacts, or priors. We provide quantitative results for 20 objects. Tac2Pose provides high accuracy pose estimations from distinctive tactile observations while regressing meaningful pose distributions to account for those contact shapes that could result from different object poses. We also test Tac2Pose on object models reconstructed from a 3D scanner, to evaluate the robustness to uncertainty in the object model. Finally, we demonstrate the advantages of Tac2Pose compared with three baseline methods for tactile pose estimation: directly regressing the object pose with a neural network, matching an observed contact to a set of possible contacts using a standard classification neural network, and direct pixel comparison of an observed contact with a set of possible contacts. Website: http://mcube.mit.edu/research/tac2pose.html