# Tactile Memory with Soft Robot: Robust Object Insertion via Masked Encoding and Soft Wrist

arXiv: https://arxiv.org/abs/2601.19275 · HTML: https://mlsanity.com/paper/2601.19275/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 49 references that are in the map.
- First posted on arXiv 27 Jan 2026.

## Builds on (most cited first, 30 of 30)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs](https://mlsanity.com/paper/1603.09320/) (2016)
- [Learning and Retrieval from Prior Data for Skill-based Imitation Learning](https://mlsanity.com/paper/2210.11435/) (2022)
- [Sparsh: Self-supervised touch representations for vision-based tactile sensing](https://mlsanity.com/paper/2410.24090/) (2024)
- [Dexterity from Touch: Self-Supervised Pre-Training of Tactile Representations with Robotic Play](https://mlsanity.com/paper/2303.12076/) (2023)
- [Behavior Retrieval: Few-Shot Imitation Learning by Querying Unlabeled Datasets](https://mlsanity.com/paper/2304.08742/) (2023)
- [Tactile-RL for Insertion: Generalization to Objects of Unknown Geometry](https://mlsanity.com/paper/2104.01167/) (2021)
- [Visuo-Tactile Transformers for Manipulation](https://mlsanity.com/paper/2210.00121/) (2022)
- [Transferable Tactile Transformers for Representation Learning Across Diverse Sensors and Tasks](https://mlsanity.com/paper/2406.13640/) (2024)
- [Masked Trajectory Models for Prediction, Representation, and Control](https://mlsanity.com/paper/2305.02968/) (2023)
- [R+X: Retrieval and Execution from Everyday Human Videos](https://mlsanity.com/paper/2407.12957/) (2024)
- [MimicTouch: Leveraging Multi-modal Human Tactile Demonstrations for Contact-rich Manipulation](https://mlsanity.com/paper/2310.16917/) (2023)
- [A Survey of Robot Manipulation in Contact](https://mlsanity.com/paper/2112.01942/) (2021)
- [Large-Scale Retrieval for Reinforcement Learning](https://mlsanity.com/paper/2206.05314/) (2022)
- [A Framework for Robot Manipulation: Skill Formalism, Meta Learning and Adaptive Control](https://mlsanity.com/paper/1805.08576/) (2018)
- [TactileGCN: A Graph Convolutional Network for Predicting Grasp Stability with Tactile Sensors](https://mlsanity.com/paper/1901.06181/) (2019)
- [On the Effectiveness of Retrieval, Alignment, and Replay in Manipulation](https://mlsanity.com/paper/2312.12345/) (2023)
- [TacGNN:Learning Tactile-based In-hand Manipulation with a Blind Robot](https://mlsanity.com/paper/2304.00736/) (2023)
- [Multi-Agent Behavior Retrieval: Retrieval-Augmented Policy Training for Cooperative Push Manipulation by Mobile Robots](https://mlsanity.com/paper/2312.02008/) (2023)
- [Contact Pose Identification for Peg-in-Hole Assembly under Uncertainties](https://mlsanity.com/paper/2101.12467/) (2021)
- [Towards Generalized Robot Assembly through Compliance-Enabled Contact Formations](https://mlsanity.com/paper/2303.05565/) (2023)
- [Learning thin deformable object manipulation with a multi-sensory integrated soft hand](https://mlsanity.com/paper/2411.13952/) (2024)
- [Towards integrated tactile sensorimotor control in anthropomorphic soft robotic hands](https://mlsanity.com/paper/2102.03318/) (2021)
- [Whisker-Inspired Tactile Sensing for Contact Localization on Robot Manipulators](https://mlsanity.com/paper/2210.12387/) (2022)
- [DexiTac: Soft Dexterous Tactile Gripping](https://mlsanity.com/paper/2405.02897/) (2024)
- [Visuotactile-Based Learning for Insertion with Compliant Hands](https://mlsanity.com/paper/2411.06408/) (2024)
- [Soft and Compliant Contact-Rich Hair Manipulation and Care](https://mlsanity.com/paper/2501.02630/) (2025)
- [Whisker-Inspired Tactile Sensing: A Sim2Real Approach for Precise Underwater Contact Tracking](https://mlsanity.com/paper/2410.14005/) (2024)

## Abstract (arXiv)

Tactile memory, the ability to store and retrieve touch-based experience, is critical for contact-rich tasks such as key insertion under uncertainty. To replicate this capability, we introduce Tactile Memory with Soft Robot (TaMeSo-bot), a system that integrates a soft wrist with tactile retrieval-based control to enable safe and robust manipulation. The soft wrist allows safe contact exploration during data collection, while tactile memory reuses past demonstrations via retrieval for flexible adaptation to unseen scenarios. The core of this system is the Masked Tactile Trajectory Transformer (MAT3), which jointly models spatiotemporal interactions between robot actions, distributed tactile cues, force-torque measurements, and proprioceptive signals. Through masked token prediction, MAT3 learns rich spatiotemporal representations by inferring missing sensory information from context, autonomously extracting task-relevant features without explicit subtask segmentation. We validate our approach on peg-in-hole tasks with diverse pegs and conditions in real-robot experiments. Our extensive evaluation demonstrates that MAT3 achieves higher success rates than the baselines over all conditions and shows remarkable capability to adapt to unseen pegs and conditions.