# Manipulation by Feel: Touch-Based Control with Deep Predictive Models

arXiv: https://arxiv.org/abs/1903.04128 · HTML: https://mlsanity.com/paper/1903.04128/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Mar 2019.

## Built on by (most cited first, 30 of 36)

- [Touch and Go: Learning from Human-Collected Vision and Touch](https://mlsanity.com/paper/2211.12498/) (2022)
- [Rotating without Seeing: Towards In-hand Dexterity through Touch](https://mlsanity.com/paper/2303.10880/) (2023)
- [Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformers](https://mlsanity.com/paper/2107.03996/) (2021)
- [The ObjectFolder Benchmark: Multisensory Learning with Neural and Real Objects](https://mlsanity.com/paper/2306.00956/) (2023)
- [GelSight Wedge: Measuring High-Resolution 3D Contact Geometry with a Compact Robot Finger](https://mlsanity.com/paper/2106.08851/) (2021)
- [SwingBot: Learning Physical Features from In-hand Tactile Exploration for Dynamic Swing-up Manipulation](https://mlsanity.com/paper/2101.11812/) (2021)
- [RoboPack: Learning Tactile-Informed Dynamics Models for Dense Packing](https://mlsanity.com/paper/2407.01418/) (2024)
- [Learning to Feel the Future: DreamTacVLA for Contact-Rich Manipulation](https://mlsanity.com/paper/2512.23864/) (2025)
- [Tac2Pose: Tactile Object Pose Estimation from the First Touch](https://mlsanity.com/paper/2204.11701/) (2022)
- [Zero-shot sim-to-real transfer of tactile control policies for aggressive swing-up manipulation](https://mlsanity.com/paper/2101.02680/) (2021)
- [Robot Skill Adaptation via Soft Actor-Critic Gaussian Mixture Models](https://mlsanity.com/paper/2111.13129/) (2021)
- [FeelAnyForce: Estimating Contact Force Feedback from Tactile Sensation for Vision-Based Tactile Sensors](https://mlsanity.com/paper/2410.02048/) (2024)
- [Sample Complexity and Overparameterization Bounds for Temporal Difference Learning with Neural Network Approximation](https://mlsanity.com/paper/2103.01391/) (2021)
- [Dynamic Modeling of Hand-Object Interactions via Tactile Sensing](https://mlsanity.com/paper/2109.04378/) (2021)
- [Action Conditioned Tactile Prediction: case study on slip prediction](https://mlsanity.com/paper/2205.09430/) (2022)
- [Controllable Visual-Tactile Synthesis](https://mlsanity.com/paper/2305.03051/) (2023)
- [M2CURL: Sample-Efficient Multimodal Reinforcement Learning via Self-Supervised Representation Learning for Robotic Manipulation](https://mlsanity.com/paper/2401.17032/) (2024)
- [The Sound of Simulation: Learning Multimodal Sim-to-Real Robot Policies with Generative Audio](https://mlsanity.com/paper/2507.02864/) (2025)
- [Elastic Tactile Simulation Towards Tactile-Visual Perception](https://mlsanity.com/paper/2108.05013/) (2021)
- [Enhancing Tactile-based Reinforcement Learning for Robotic Control](https://mlsanity.com/paper/2510.21609/) (2025)
- [LeTac-MPC: Learning Model Predictive Control for Tactile-reactive Grasping](https://mlsanity.com/paper/2403.04934/) (2024)
- [DexSkin: High-Coverage Conformable Robotic Skin for Learning Contact-Rich Manipulation](https://mlsanity.com/paper/2509.18830/) (2025)
- [Tactile-Sensitive NewtonianVAE for High-Accuracy Industrial Connector Insertion](https://mlsanity.com/paper/2203.05955/) (2022)
- [A Differentiable Recipe for Learning Visual Non-Prehensile Planar Manipulation](https://mlsanity.com/paper/2111.05318/) (2021)
- [Bench2Dex: Benchmarking Visuo-Tactile Bimanual Dexterous Manipulation Across Dexterous Hands](https://mlsanity.com/paper/2609.15726/) (2026)
- [OpenBot-Fleet: A System for Collective Learning with Real Robots](https://mlsanity.com/paper/2405.07515/) (2024)
- [CoStream: Composing Simple Behaviors for Generalizable Complex Manipulation](https://mlsanity.com/paper/2606.26423/) (2026)
- [Deep Functional Predictive Control for Strawberry Cluster Manipulation using Tactile Prediction](https://mlsanity.com/paper/2303.05393/) (2023)
- [Multi-Modal World Model for Physical Robot Interactions: Simultaneous Visual and Tactile Predictions for Enhanced Accuracy](https://mlsanity.com/paper/2304.11193/) (2023)
- [UniTac2Pose: A Unified Approach Learned in Simulation for Category-level Visuotactile In-hand Pose Estimation](https://mlsanity.com/paper/2509.15934/) (2025)

## Abstract (arXiv)

Touch sensing is widely acknowledged to be important for dexterous robotic manipulation, but exploiting tactile sensing for continuous, non-prehensile manipulation is challenging. General purpose control techniques that are able to effectively leverage tactile sensing as well as accurate physics models of contacts and forces remain largely elusive, and it is unclear how to even specify a desired behavior in terms of tactile percepts. In this paper, we take a step towards addressing these issues by combining high-resolution tactile sensing with data-driven modeling using deep neural network dynamics models. We propose deep tactile MPC, a framework for learning to perform tactile servoing from raw tactile sensor inputs, without manual supervision. We show that this method enables a robot equipped with a GelSight-style tactile sensor to manipulate a ball, analog stick, and 20-sided die, learning from unsupervised autonomous interaction and then using the learned tactile predictive model to reposition each object to user-specified configurations, indicated by a goal tactile reading. Videos, visualizations and the code are available here: https://sites.google.com/view/deeptactilempc