# Dynamic Modeling of Hand-Object Interactions via Tactile Sensing arXiv: https://arxiv.org/abs/2109.04378 ยท HTML: https://mlsanity.com/paper/2109.04378/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Builds on 37 of its 71 references that are in the map. - First posted on arXiv 9 Sep 2021. ## Builds on (most cited first, 30 of 37) - [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014) - [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018) - [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015) - [Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination](https://mlsanity.com/paper/1805.01978/) (2018) - [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015) - [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014) - [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019) - [Learning Dexterous In-Hand Manipulation](https://mlsanity.com/paper/1808.00177/) (2018) - [Learning joint reconstruction of hands and manipulated objects](https://mlsanity.com/paper/1904.05767/) (2019) - [Deep Dynamics Models for Learning Dexterous Manipulation](https://mlsanity.com/paper/1909.11652/) (2019) - [ContactPose: A Dataset of Grasps with Object Contact and Hand Pose](https://mlsanity.com/paper/2007.09545/) (2020) - [Deep Learning Face Representation by Joint Identification-Verification](https://mlsanity.com/paper/1406.4773/) (2014) - [Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks](https://mlsanity.com/paper/1810.10191/) (2018) - [ContactDB: Analyzing and Predicting Grasp Contact via Thermal Imaging](https://mlsanity.com/paper/1904.06830/) (2019) - [The Sound of Motions](https://mlsanity.com/paper/1904.05979/) (2019) - [Zero-Shot Learning by Convex Combination of Semantic Embeddings](https://mlsanity.com/paper/1312.5650/) (2013) - [Capturing Hands in Action using Discriminative Salient Points and Physics Simulation](https://mlsanity.com/paper/1506.02178/) (2015) - [Foley Music: Learning to Generate Music from Videos](https://mlsanity.com/paper/2007.10984/) (2020) - [Ambient Sound Provides Supervision for Visual Learning](https://mlsanity.com/paper/1608.07017/) (2016) - [Connecting Touch and Vision via Cross-Modal Prediction](https://mlsanity.com/paper/1906.06322/) (2019) - [Real-time Joint Tracking of a Hand Manipulating an Object from RGB-D Input](https://mlsanity.com/paper/1610.04889/) (2016) - [Cable Manipulation with a Tactile-Reactive Gripper](https://mlsanity.com/paper/1910.02860/) (2019) - [Discriminative Correlation Filter with Channel and Spatial Reliability](https://mlsanity.com/paper/1611.08461/) (2016) - [Use the Force, Luke! Learning to Predict Physical Forces by Simulating Effects](https://mlsanity.com/paper/2003.12045/) (2020) - [Manipulation by Feel: Touch-Based Control with Deep Predictive Models](https://mlsanity.com/paper/1903.04128/) (2019) - [Connecting Look and Feel: Associating the visual and tactile properties of physical materials](https://mlsanity.com/paper/1704.03822/) (2017) - [Shape-independent Hardness Estimation Using Deep Learning and a GelSight Tactile Sensor](https://mlsanity.com/paper/1704.03955/) (2017) - [Tactile Mapping and Localization from High-Resolution Tactile Imprints](https://mlsanity.com/paper/1904.10944/) (2019) - [Embedding Label Structures for Fine-Grained Feature Representation](https://mlsanity.com/paper/1512.02895/) (2015) - [Learning Joint Representations of Videos and Sentences with Web Image Search](https://mlsanity.com/paper/1608.02367/) (2016) ## Built on by (most cited first, 8 of 8) - [Touch and Go: Learning from Human-Collected Vision and Touch](https://mlsanity.com/paper/2211.12498/) (2022) - [Controllable Visual-Tactile Synthesis](https://mlsanity.com/paper/2305.03051/) (2023) - [Dynamic Reconstruction of Hand-Object Interaction with Distributed Force-aware Contact Representation](https://mlsanity.com/paper/2411.09572/) (2024) - [TactoFind: A Tactile Only System for Object Retrieval](https://mlsanity.com/paper/2303.13482/) (2023) - [Capturing complex hand movements and object interactions using machine learning-powered stretchable smart textile gloves](https://mlsanity.com/paper/2410.02221/) (2024) - [Intelligent Knee Sleeves: A Real-time Multimodal Dataset for 3D Lower Body Motion Estimation Using Smart Textile](https://mlsanity.com/paper/2311.12829/) (2023) - [MassNet: A Deep Learning Approach for Body Weight Extraction from A Single Pressure Image](https://mlsanity.com/paper/2303.10136/) (2023) - [BiView-Touch: Learning Bimanual Tactile Representations by Cross-Hand Completion](https://mlsanity.com/paper/2609.23352/) (2026) ## Abstract (arXiv) Tactile sensing is critical for humans to perform everyday tasks. While significant progress has been made in analyzing object grasping from vision, it remains unclear how we can utilize tactile sensing to reason about and model the dynamics of hand-object interactions. In this work, we employ a high-resolution tactile glove to perform four different interactive activities on a diversified set of objects. We build our model on a cross-modal learning framework and generate the labels using a visual processing pipeline to supervise the tactile model, which can then be used on its own during the test time. The tactile model aims to predict the 3d locations of both the hand and the object purely from the touch data by combining a predictive model and a contrastive learning module. This framework can reason about the interaction patterns from the tactile data, hallucinate the changes in the environment, estimate the uncertainty of the prediction, and generalize to unseen objects. We also provide detailed ablation studies regarding different system designs as well as visualizations of the predicted trajectories. This work takes a step on dynamics modeling in hand-object interactions from dense tactile sensing, which opens the door for future applications in activity learning, human-computer interactions, and imitation learning for robotics.