# ViTacGen: Robotic Pushing with Vision-to-Touch Generation

arXiv: https://arxiv.org/abs/2510.14117 · HTML: https://mlsanity.com/paper/2510.14117/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods.
- Builds on 28 of its 39 references that are in the map.
- First posted on arXiv 15 Oct 2025.

## Placed in

- Deep reinforcement learning methods / [RL for robot locomotion and manipulation](https://mlsanity.com/deeprl/rl-for-robot-locomotion-and-manipulation/)

## Builds on (most cited first, 28 of 28)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [Touch and Go: Learning from Human-Collected Vision and Touch](https://mlsanity.com/paper/2211.12498/) (2022)
- [GelSlim: A High-Resolution, Compact, Robust, and Calibrated Tactile-sensing Finger](https://mlsanity.com/paper/1803.00628/) (2018)
- [Visuo-Tactile Transformers for Manipulation](https://mlsanity.com/paper/2210.00121/) (2022)
- [More than a Million Ways to Be Pushed: A High-Fidelity Experimental Dataset of Planar Pushing](https://mlsanity.com/paper/1604.04038/) (2016)
- [ViTac: Feature Sharing between Vision and Tactile Sensing for Cloth Texture Recognition](https://mlsanity.com/paper/1802.07490/) (2018)
- [Robotic Tactile Perception of Object Properties: A Review](https://mlsanity.com/paper/1711.03810/) (2017)
- [Tactile Sim-to-Real Policy Transfer via Real-to-Sim Image Translation](https://mlsanity.com/paper/2106.08796/) (2021)
- [Tactile Gym 2.0: Sim-to-real Deep Reinforcement Learning for Comparing Low-cost High-Resolution Robot Touch](https://mlsanity.com/paper/2207.10763/) (2022)
- ["Touching to See" and "Seeing to Feel": Robotic Cross-modal SensoryData Generation for Visual-Tactile Perception](https://mlsanity.com/paper/1902.06273/) (2019)
- [Bi-Touch: Bimanual Tactile Manipulation with Sim-to-Real Deep Reinforcement Learning](https://mlsanity.com/paper/2307.06423/) (2023)
- [S3M-Net: Joint Learning of Semantic Segmentation and Stereo Matching for Autonomous Driving](https://mlsanity.com/paper/2401.11414/) (2024)
- [Spatio-temporal Attention Model for Tactile Texture Recognition](https://mlsanity.com/paper/2008.04442/) (2020)
- [Sim-to-Real Model-Based and Model-Free Deep Reinforcement Learning for Tactile Pushing](https://mlsanity.com/paper/2307.14272/) (2023)
- [Goal-Driven Robotic Pushing Using Tactile and Proprioceptive Feedback](https://mlsanity.com/paper/2012.01859/) (2020)
- [M2CURL: Sample-Efficient Multimodal Reinforcement Learning via Self-Supervised Representation Learning for Robotic Manipulation](https://mlsanity.com/paper/2401.17032/) (2024)
- [ConViTac: Aligning Visual-Tactile Fusion with Contrastive Representations](https://mlsanity.com/paper/2506.20757/) (2025)
- [Vis2Hap: Vision-based Haptic Rendering by Cross-modal Generation](https://mlsanity.com/paper/2301.06826/) (2023)
- [CDI3D: Cross-guided Dense-view Interpolation for 3D Reconstruction](https://mlsanity.com/paper/2503.08005/) (2025)

## Built on by (most cited first, 3 of 3)

- [Learning to Feel the Future: DreamTacVLA for Contact-Rich Manipulation](https://mlsanity.com/paper/2512.23864/) (2025)
- [FlowTouch: View-Invariant Visuo-Tactile Prediction](https://mlsanity.com/paper/2603.08255/) (2026)
- [VTLoc: Learning-based Tactile Contact Localization in Visual Point Clouds](https://mlsanity.com/paper/2607.16146/) (2026)

## Abstract (arXiv)

Robotic pushing is a fundamental manipulation task that requires tactile feedback to capture subtle contact forces and dynamics between the end-effector and the object. However, real tactile sensors often face hardware limitations such as high costs and fragility, and deployment challenges involving calibration and variations between different sensors, while vision-only policies struggle with satisfactory performance. Inspired by humans' ability to infer tactile states from vision, we propose ViTacGen, a novel robot manipulation framework designed for visual robotic pushing with vision-to-touch generation in reinforcement learning to eliminate the reliance on high-resolution real tactile sensors, enabling effective zero-shot deployment on visual-only robotic systems. Specifically, ViTacGen consists of an encoder-decoder vision-to-touch generation network that generates contact depth images, a standardized tactile representation, directly from visual image sequence, followed by a reinforcement learning policy that fuses visual-tactile data with contrastive learning based on visual and generated tactile observations. We validate the effectiveness of our approach in both simulation and real world experiments, demonstrating its superior performance and achieving a success rate of up to 86%.