# Tactile-Filter: Interactive Tactile Perception for Part Mating

arXiv: https://arxiv.org/abs/2303.06034 · HTML: https://mlsanity.com/paper/2303.06034/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 43 references that are in the map.
- First posted on arXiv 10 Mar 2023.

## Builds on (most cited first, 30 of 33)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Where2Act: From Pixels to Actions for Articulated 3D Objects](https://mlsanity.com/paper/2101.02692/) (2021)
- [BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environments](https://mlsanity.com/paper/2108.03332/) (2021)
- [Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks](https://mlsanity.com/paper/1810.10191/) (2018)
- [More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch](https://mlsanity.com/paper/1805.11085/) (2018)
- [Interactive Gibson Benchmark (iGibson 0.5): A Benchmark for Interactive Navigation in Cluttered Environments](https://mlsanity.com/paper/1910.14442/) (2019)
- [VAT-Mart: Learning Visual Action Trajectory Proposals for Manipulating 3D ARTiculated Objects](https://mlsanity.com/paper/2106.14440/) (2021)
- [Interactive Perception: Leveraging Action in Perception and Perception in Action](https://mlsanity.com/paper/1604.03670/) (2016)
- [TACTO: A Fast, Flexible, and Open-source Simulator for High-Resolution Vision-based Tactile Sensors](https://mlsanity.com/paper/2012.08456/) (2020)
- [An End-to-End Differentiable Framework for Contact-Aware Robot Design](https://mlsanity.com/paper/2107.07501/) (2021)
- [Tactile-RL for Insertion: Generalization to Objects of Unknown Geometry](https://mlsanity.com/paper/2104.01167/) (2021)
- [Cable Manipulation with a Tactile-Reactive Gripper](https://mlsanity.com/paper/1910.02860/) (2019)
- [GelSlim3.0: High-Resolution Measurement of Shape, Force and Slip in a Compact Tactile-Sensing Finger](https://mlsanity.com/paper/2103.12269/) (2021)
- [MidasTouch: Monte-Carlo inference over distributions across sliding touch](https://mlsanity.com/paper/2210.14210/) (2022)
- [Active Clothing Material Perception using Tactile Sensing and Deep Learning](https://mlsanity.com/paper/1711.00574/) (2017)
- [Tactile Dexterity: Manipulation Primitives with Tactile Feedback](https://mlsanity.com/paper/2002.03236/) (2020)
- [Tactile Sim-to-Real Policy Transfer via Real-to-Sim Image Translation](https://mlsanity.com/paper/2106.08796/) (2021)
- [Object Finding in Cluttered Scenes Using Interactive Perception](https://mlsanity.com/paper/1911.07482/) (2019)
- [Maintaining Grasps within Slipping Bound by Monitoring Incipient Slip](https://mlsanity.com/paper/1810.13381/) (2018)
- [Tac2Pose: Tactile Object Pose Estimation from the First Touch](https://mlsanity.com/paper/2204.11701/) (2022)
- [Extrinsic Contact Sensing with Relative-Motion Tracking from Distributed Tactile Measurements](https://mlsanity.com/paper/2103.08108/) (2021)
- [Tactile-Based Insertion for Dense Box-Packing](https://mlsanity.com/paper/1909.05426/) (2019)
- [Localizing the Object Contact through Matching Tactile Features with Visual Map](https://mlsanity.com/paper/1708.04441/) (2017)
- [Collision-aware In-hand 6D Object Pose Estimation using Multiple Vision-based Tactile Sensors](https://mlsanity.com/paper/2301.13667/) (2023)
- [Interpreting Contact Interactions to Overcome Failure in Robot Assembly Tasks](https://mlsanity.com/paper/2101.02725/) (2021)
- [Tactile Pose Estimation and Policy Learning for Unknown Object Manipulation](https://mlsanity.com/paper/2203.10685/) (2022)
- [Imitation and Supervised Learning of Compliance for Robotic Assembly](https://mlsanity.com/paper/2111.10488/) (2021)

## Built on by (most cited first, 4 of 4)

- [Robust Pivoting Manipulation using Contact Implicit Bilevel Optimization](https://mlsanity.com/paper/2303.08965/) (2023)
- [Tactile Estimation of Extrinsic Contact Patch for Stable Placement](https://mlsanity.com/paper/2309.14552/) (2023)
- [Zero-Shot Peg Insertion: Identifying Mating Holes and Estimating SE(2) Poses with Vision-Language Models](https://mlsanity.com/paper/2503.06026/) (2025)
- [ControlTac: Scaling Tactile Data with Physically Controlled Tactile Image Generation](https://mlsanity.com/paper/2505.20498/) (2025)

## Abstract (arXiv)

Humans rely on touch and tactile sensing for a lot of dexterous manipulation tasks. Our tactile sensing provides us with a lot of information regarding contact formations as well as geometric information about objects during any interaction. With this motivation, vision-based tactile sensors are being widely used for various robotic perception and control tasks. In this paper, we present a method for interactive perception using vision-based tactile sensors for a part mating task, where a robot can use tactile sensors and a feedback mechanism using a particle filter to incrementally improve its estimate of objects (pegs and holes) that fit together. To do this, we first train a deep neural network that makes use of tactile images to predict the probabilistic correspondence between arbitrarily shaped objects that fit together. The trained model is used to design a particle filter which is used twofold. First, given one partial (or non-unique) observation of the hole, it incrementally improves the estimate of the correct peg by sampling more tactile observations. Second, it selects the next action for the robot to sample the next touch (and thus image) which results in maximum uncertainty reduction to minimize the number of interactions during the perception task. We evaluate our method on several part-mating tasks with novel objects using a robot equipped with a vision-based tactile sensor. We also show the efficiency of the proposed action selection method against a naive method. See supplementary video at https://www.youtube.com/watch?v=jMVBg_e3gLw .