# LeTac-MPC: Learning Model Predictive Control for Tactile-reactive Grasping arXiv: https://arxiv.org/abs/2403.04934 ยท HTML: https://mlsanity.com/paper/2403.04934/ ## Key facts - Cited by 4 papers in the map (data through 14 Oct 2026). - Builds on 39 of its 53 references that are in the map. - First posted on arXiv 7 Mar 2024. ## Builds on (most cited first, 30 of 39) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021) - [OptNet: Differentiable Optimization as a Layer in Neural Networks](https://mlsanity.com/paper/1703.00443/) (2017) - [DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation](https://mlsanity.com/paper/2005.14679/) (2020) - [OSQP: An Operator Splitting Solver for Quadratic Programs](https://mlsanity.com/paper/1711.08013/) (2017) - [TossingBot: Learning to Throw Arbitrary Objects with Residual Physics](https://mlsanity.com/paper/1903.11239/) (2019) - [3D Neural Scene Representations for Visuomotor Control](https://mlsanity.com/paper/2107.04004/) (2021) - [More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch](https://mlsanity.com/paper/1805.11085/) (2018) - [The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?](https://mlsanity.com/paper/1710.05512/) (2017) - [Learning Multi-Object Dynamics with Compositional Neural Radiance Fields](https://mlsanity.com/paper/2202.11855/) (2022) - [GelSlim: A High-Resolution, Compact, Robust, and Calibrated Tactile-sensing Finger](https://mlsanity.com/paper/1803.00628/) (2018) - [Cable Manipulation with a Tactile-Reactive Gripper](https://mlsanity.com/paper/1910.02860/) (2019) - [Iterative Residual Policy: for Goal-Conditioned Dynamic Manipulation of Deformable Objects](https://mlsanity.com/paper/2203.00663/) (2022) - [Manipulation by Feel: Touch-Based Control with Deep Predictive Models](https://mlsanity.com/paper/1903.04128/) (2019) - [GelSight Wedge: Measuring High-Resolution 3D Contact Geometry with a Compact Robot Finger](https://mlsanity.com/paper/2106.08851/) (2021) - [Active Clothing Material Perception using Tactile Sensing and Deep Learning](https://mlsanity.com/paper/1711.00574/) (2017) - [ViTac: Feature Sharing between Vision and Tactile Sensing for Cloth Texture Recognition](https://mlsanity.com/paper/1802.07490/) (2018) - [Robotic Tactile Perception of Object Properties: A Review](https://mlsanity.com/paper/1711.03810/) (2017) - [SwingBot: Learning Physical Features from In-hand Tactile Exploration for Dynamic Swing-up Manipulation](https://mlsanity.com/paper/2101.11812/) (2021) - [Tactile Dexterity: Manipulation Primitives with Tactile Feedback](https://mlsanity.com/paper/2002.03236/) (2020) - [Visuotactile Affordances for Cloth Manipulation with Local Control](https://mlsanity.com/paper/2212.05108/) (2022) - [Shape-independent Hardness Estimation Using Deep Learning and a GelSight Tactile Sensor](https://mlsanity.com/paper/1704.03955/) (2017) - [Active Extrinsic Contact Sensing: Application to General Peg-in-Hole Insertion](https://mlsanity.com/paper/2110.03555/) (2021) - [Dense Tactile Force Distribution Estimation using GelSlim and inverse FEM](https://mlsanity.com/paper/1810.04621/) (2018) - [Maintaining Grasps within Slipping Bound by Monitoring Incipient Slip](https://mlsanity.com/paper/1810.13381/) (2018) - [Zero-shot sim-to-real transfer of tactile control policies for aggressive swing-up manipulation](https://mlsanity.com/paper/2101.02680/) (2021) - [Tactile Regrasp: Grasp Adjustments via Simulated Tactile Transformations](https://mlsanity.com/paper/1803.01940/) (2018) - [Learning Generalizable Vision-Tactile Robotic Grasping Strategy for Deformable Objects via Transformer](https://mlsanity.com/paper/2112.06374/) (2021) - [DTact: A Vision-Based Tactile Sensor that Measures High-Resolution 3D Geometry Directly from Darkness](https://mlsanity.com/paper/2209.13916/) (2022) - [Goal-Driven Robotic Pushing Using Tactile and Proprioceptive Feedback](https://mlsanity.com/paper/2012.01859/) (2020) ## Built on by (most cited first, 4 of 4) - [Bench2Dex: Benchmarking Visuo-Tactile Bimanual Dexterous Manipulation Across Dexterous Hands](https://mlsanity.com/paper/2609.15726/) (2026) - [OmniTacTune: Policy-Agnostic Real-World RL for Tactile Residual Adaptation of Visual Policies](https://mlsanity.com/paper/2607.03723/) (2026) - [Symmetry-Aware Fusion of Vision and Tactile Sensing via Bilateral Force Priors for Robotic Manipulation](https://mlsanity.com/paper/2602.13689/) (2026) - [TACTIC: Tactile and Vision Conditioned Contact-Centric Control for Whole-Arm Manipulation](https://mlsanity.com/paper/2607.09218/) (2026) ## Abstract (arXiv) Grasping is a crucial task in robotics, necessitating tactile feedback and reactive grasping adjustments for robust grasping of objects under various conditions and with differing physical properties. In this paper, we introduce LeTac-MPC, a learning-based model predictive control (MPC) for tactile-reactive grasping. Our approach enables the gripper to grasp objects with different physical properties on dynamic and force-interactive tasks. We utilize a vision-based tactile sensor, GelSight, which is capable of perceiving high-resolution tactile feedback that contains information on the physical properties and states of the grasped object. LeTac-MPC incorporates a differentiable MPC layer designed to model the embeddings extracted by a neural network (NN) from tactile feedback. This design facilitates convergent and robust grasping control at a frequency of 25 Hz. We propose a fully automated data collection pipeline and collect a dataset only using standardized blocks with different physical properties. However, our trained controller can generalize to daily objects with different sizes, shapes, materials, and textures. The experimental results demonstrate the effectiveness and robustness of the proposed approach. We compare LeTac-MPC with two purely model-based tactile-reactive controllers (MPC and PD) and open-loop grasping. Our results show that LeTac-MPC has optimal performance in dynamic and force-interactive tasks and optimal generalizability. We release our code and dataset at https://github.com/ZhengtongXu/LeTac-MPC.