# ViTaSCOPE: Visuo-tactile Implicit Representation for In-hand Pose and Extrinsic Contact Estimation

arXiv: https://arxiv.org/abs/2506.12239 · HTML: https://mlsanity.com/paper/2506.12239/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 46 references that are in the map.
- First posted on arXiv 13 Jun 2025.

## Builds on (most cited first, 30 of 35)

- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning](https://mlsanity.com/paper/2108.10470/) (2021)
- [Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks](https://mlsanity.com/paper/2401.14159/) (2024)
- [FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects](https://mlsanity.com/paper/2312.08344/) (2023)
- [DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation](https://mlsanity.com/paper/2005.14679/) (2020)
- [What's in your hands? 3D Reconstruction of Generic Objects in Hands](https://mlsanity.com/paper/2204.07153/) (2022)
- [TACTO: A Fast, Flexible, and Open-source Simulator for High-Resolution Vision-based Tactile Sensors](https://mlsanity.com/paper/2012.08456/) (2020)
- [AlignSDF: Pose-Aligned Signed Distance Fields for Hand-Object Reconstruction](https://mlsanity.com/paper/2207.12909/) (2022)
- [Taxim: An Example-based Simulation Model for GelSight Tactile Sensors](https://mlsanity.com/paper/2109.04027/) (2021)
- [3D Shape Reconstruction from Vision and Touch](https://mlsanity.com/paper/2007.03778/) (2020)
- [GelSlim3.0: High-Resolution Measurement of Shape, Force and Slip in a Compact Tactile-Sensing Finger](https://mlsanity.com/paper/2103.12269/) (2021)
- [Active 3D Shape Reconstruction from Vision and Touch](https://mlsanity.com/paper/2107.09584/) (2021)
- [TacSL: A Library for Visuotactile Sensor Simulation and Learning](https://mlsanity.com/paper/2408.06506/) (2024)
- [The Power of the Senses: Generalizable Manipulation from Vision and Touch through Masked Multimodal Learning](https://mlsanity.com/paper/2311.00924/) (2023)
- [DIFFTACTILE: A Physics-based Differentiable Tactile Simulator for Contact-rich Robotic Manipulation](https://mlsanity.com/paper/2403.08716/) (2024)
- [VIRDO: Visio-tactile Implicit Representations of Deformable Objects](https://mlsanity.com/paper/2202.00868/) (2022)
- [Digitizing Touch with an Artificial Multimodal Fingertip](https://mlsanity.com/paper/2411.02479/) (2024)
- [Dense Tactile Force Distribution Estimation using GelSlim and inverse FEM](https://mlsanity.com/paper/1810.04621/) (2018)
- [Soft-bubble: A highly compliant dense geometry tactile sensor for robot manipulation](https://mlsanity.com/paper/1904.02252/) (2019)
- [Tac2Pose: Tactile Object Pose Estimation from the First Touch](https://mlsanity.com/paper/2204.11701/) (2022)
- [ViHOPE: Visuotactile In-Hand Object 6D Pose Estimation with Shape Completion](https://mlsanity.com/paper/2309.05662/) (2023)
- [Extrinsic Contact Sensing with Relative-Motion Tracking from Distributed Tactile Measurements](https://mlsanity.com/paper/2103.08108/) (2021)
- [VIRDO++: Real-World, Visuo-tactile Dynamics and Perception of Deformable Objects](https://mlsanity.com/paper/2210.03701/) (2022)
- [Neural Contact Fields: Tracking Extrinsic Contact with Tactile Sensing](https://mlsanity.com/paper/2210.09297/) (2022)
- [Tactile-Based Insertion for Dense Box-Packing](https://mlsanity.com/paper/1909.05426/) (2019)
- [Manipulation via Membranes: High-Resolution and Highly Deformable Tactile Sensing and Control](https://mlsanity.com/paper/2209.13432/) (2022)
- [Simultaneous Tactile Estimation and Control of Extrinsic Contact](https://mlsanity.com/paper/2303.03385/) (2023)
- [Tactile-Driven Non-Prehensile Object Manipulation via Extrinsic Contact Mode Control](https://mlsanity.com/paper/2405.18214/) (2024)
- [Perceiving Extrinsic Contacts from Touch Improves Learning Insertion Policies](https://mlsanity.com/paper/2309.16652/) (2023)
- [TEXterity: Tactile Extrinsic deXterity](https://mlsanity.com/paper/2401.10230/) (2024)

## Built on by (most cited first, 3 of 3)

- [HydroShear: Hydroelastic Shear Simulation for Tactile Sim-to-Real Reinforcement Learning](https://mlsanity.com/paper/2603.00446/) (2026)
- [Blind Dexterous Grasping via Real2Sim2Real Tactile Policy Learning](https://mlsanity.com/paper/2606.11767/) (2026)
- [UNIC: Learning Unified Multimodal Extrinsic Contact Estimation](https://mlsanity.com/paper/2601.04356/) (2026)

## Abstract (arXiv)

Mastering dexterous, contact-rich object manipulation demands precise estimation of both in-hand object poses and external contact locationsunicodex2013tasks particularly challenging due to partial and noisy observations. We present ViTaSCOPE: Visuo-Tactile Simultaneous Contact and Object Pose Estimation, an object-centric neural implicit representation that fuses vision and high-resolution tactile feedback. By representing objects as signed distance fields and distributed tactile feedback as neural shear fields, ViTaSCOPE accurately localizes objects and registers extrinsic contacts onto their 3D geometry as contact fields. Our method enables seamless reasoning over complementary visuo-tactile cues by leveraging simulation for scalable training and zero-shot transfers to the real-world by bridging the sim-to-real gap. We evaluate our method through comprehensive simulated and real-world experiments, demonstrating its capabilities in dexterous manipulation scenarios.