# Tac2Pix: Image-Space Visuo-Tactile Fusion for Dexterous Manipulation

arXiv: https://arxiv.org/abs/2606.08765 · HTML: https://mlsanity.com/paper/2606.08765/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 47 of its 53 references that are in the map.
- First posted on arXiv 7 Jun 2026.

## Builds on (most cited first, 30 of 47)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [R3M: A Universal Visual Representation for Robot Manipulation](https://mlsanity.com/paper/2203.12601/) (2022)
- [SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics](https://mlsanity.com/paper/2506.01844/) (2025)
- [Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation](https://mlsanity.com/paper/1710.04615/) (2017)
- [Real-World Robot Learning with Masked Visual Pre-training](https://mlsanity.com/paper/2210.03109/) (2022)
- [Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence?](https://mlsanity.com/paper/2303.18240/) (2023)
- [DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation](https://mlsanity.com/paper/2005.14679/) (2020)
- [Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks](https://mlsanity.com/paper/1810.10191/) (2018)
- [More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch](https://mlsanity.com/paper/1805.11085/) (2018)
- [LEAP Hand: Low-Cost, Efficient, and Anthropomorphic Hand for Robot Learning](https://mlsanity.com/paper/2309.06440/) (2023)
- [LeRobot: An Open-Source Library for End-to-End Robot Learning](https://mlsanity.com/paper/2602.22818/) (2026)
- [3D-ViTac: Learning Fine-Grained Manipulation with Visuo-Tactile Sensing](https://mlsanity.com/paper/2410.24091/) (2024)
- [The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?](https://mlsanity.com/paper/1710.05512/) (2017)
- [Dexterity from Touch: Self-Supervised Pre-Training of Tactile Representations with Robotic Play](https://mlsanity.com/paper/2303.12076/) (2023)
- [Learning Generalizable Dexterous Manipulation from Human Grasp Affordance](https://mlsanity.com/paper/2204.02320/) (2022)
- [AnyTouch: Learning Unified Static-Dynamic Representation across Multiple Visuo-tactile Sensors](https://mlsanity.com/paper/2502.12191/) (2025)
- [Visuo-Tactile Transformers for Manipulation](https://mlsanity.com/paper/2210.00121/) (2022)
- [Cable Manipulation with a Tactile-Reactive Gripper](https://mlsanity.com/paper/1910.02860/) (2019)
- [Multimodal Visual-Tactile Representation Learning through Self-Supervised Contrastive Pre-Training](https://mlsanity.com/paper/2401.12024/) (2024)
- [ARCap: Collecting High-quality Human Demonstrations for Robot Learning with Augmented Reality Feedback](https://mlsanity.com/paper/2410.08464/) (2024)
- [Touch in the Wild: Learning Fine-Grained Manipulation with a Portable Visuo-Tactile Gripper](https://mlsanity.com/paper/2507.15062/) (2025)

## Abstract (arXiv)

Vision and touch are naturally complementary for dexterous manipulation. Yet they are represented in fundamentally different forms: visual observations are dense and spatially structured, whereas tactile measurements are sparse, heterogeneous, and robot-centric. This mismatch makes it challenging to establish explicit spatial correspondence between the two modalities, particularly when they are fused in latent space. However, each tactile measurement is associated with a known location on the robot's kinematic chain, offering a localization prior that could bridge this representational gap. This raises a natural question: can this localization prior be used to construct a shared representation that explicitly aligns touch with vision? We introduce Tac2Pix, an image-space visuo-tactile fusion framework that projects sparse tactile measurements onto the RGB image plane using forward kinematics and camera calibration, then renders them as force-aware saliency maps. By expressing touch in the same pixel space as vision, Tac2Pix directly leverages pretrained 2D visual representations. We further employ zero-initialized input adaptation to preserve the original visual pathway at the start of policy learning. To assess whether this geometry-based interface yields consistent policy gains, we evaluate Tac2Pix on three simulated and three real-world dexterous manipulation tasks. In simulation, Tac2Pix consistently improves visuo-tactile policy learning across tasks, policy architectures, and visual backbones. In real-world experiments, it improves average success under visual occlusion by 26.7 percentage points over the strongest latent visuo-tactile fusion baseline, with gains persisting under both random and physical occlusions. Together, these results support image space as an effective interface for fusing sparse touch with pretrained visual representations.