# Transparent Fragments Contour Estimation via Visual-Tactile Fusion for Autonomous Reassembly

arXiv: https://arxiv.org/abs/2603.20290 · HTML: https://mlsanity.com/paper/2603.20290/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 32 of its 46 references that are in the map.
- First posted on arXiv 18 Mar 2026.

## Builds on (most cited first, 30 of 32)

- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulation](https://mlsanity.com/paper/2005.14679/) (2020)
- [ClearGrasp: 3D Shape Estimation of Transparent Objects for Manipulation](https://mlsanity.com/paper/1910.02550/) (2019)
- [Better Aggregation in Test-Time Augmentation](https://mlsanity.com/paper/2011.11156/) (2020)
- [Segmenting Transparent Objects in the Wild](https://mlsanity.com/paper/2003.13948/) (2020)
- [KeyPose: Multi-View 3D Labeling and Keypoint Estimation for Transparent Objects](https://mlsanity.com/paper/1912.02805/) (2019)
- [RGB-D Local Implicit Function for Depth Completion of Transparent Objects](https://mlsanity.com/paper/2104.00622/) (2021)
- [Breaking Bad: A Dataset for Geometric Fracture and Reassembly](https://mlsanity.com/paper/2210.11463/) (2022)
- [GelSight Wedge: Measuring High-Resolution 3D Contact Geometry with a Compact Robot Finger](https://mlsanity.com/paper/2106.08851/) (2021)
- [DiffAssemble: A Unified Graph-Diffusion Model for 2D and 3D Reassembly](https://mlsanity.com/paper/2402.19302/) (2024)
- [Jigsaw: Learning to Assemble Multiple Fractured Objects](https://mlsanity.com/paper/2305.17975/) (2023)
- [Robotic Perception of Transparent Objects: A Review](https://mlsanity.com/paper/2304.00157/) (2023)
- [Leveraging SE(3) Equivariance for Learning 3D Geometric Shape Assembly](https://mlsanity.com/paper/2309.06810/) (2023)
- [TransCut: Transparent Object Segmentation from a Light-Field Image](https://mlsanity.com/paper/1511.06853/) (2015)
- [Glass Segmentation with RGB-Thermal Image Pairs](https://mlsanity.com/paper/2204.05453/) (2022)
- [StereoPose: Category-Level 6D Transparent Object Pose Estimation from Stereo Images via Back-View NOCS](https://mlsanity.com/paper/2211.01644/) (2022)
- [GelTip: A Finger-shaped Optical Tactile Sensor for Robotic Manipulation](https://mlsanity.com/paper/2008.05404/) (2020)
- [MVTrans: Multi-View Perception of Transparent Objects](https://mlsanity.com/paper/2302.11683/) (2023)
- [GlassLoc: Plenoptic Grasp Pose Detection in Transparent Clutter](https://mlsanity.com/paper/1909.04269/) (2019)
- [A New Benchmark: On the Utility of Synthetic Data with Blender for Bare Supervised Learning and Downstream Domain Adaptation](https://mlsanity.com/paper/2303.09165/) (2023)
- [Polarimetric Inverse Rendering for Transparent Shapes Reconstruction](https://mlsanity.com/paper/2208.11836/) (2022)
- [Large-Field Contextual Feature Learning for Glass Detection](https://mlsanity.com/paper/2209.04639/) (2022)
- [A Flexible and Robust Large Scale Capacitive Tactile System for Robots](https://mlsanity.com/paper/1411.6837/) (2014)
- [Touch if it's transparent! ACTOR: Active Tactile-based Category-Level Transparent Object Reconstruction](https://mlsanity.com/paper/2307.16254/) (2023)
- [SuperCaustics: Real-time, open-source simulation of transparent objects for deep learning applications](https://mlsanity.com/paper/2107.11008/) (2021)
- [Visual-tactile Fusion for Transparent Object Grasping in Complex Backgrounds](https://mlsanity.com/paper/2211.16693/) (2022)
- [Where Shall I Touch? Vision-Guided Tactile Poking for Transparent Object Grasping](https://mlsanity.com/paper/2208.09743/) (2022)
- [Learning to Assemble Geometric Shapes](https://mlsanity.com/paper/2205.11809/) (2022)

## Abstract (arXiv)

The contour estimation of transparent fragments is very important for autonomous reassembly, especially in the fields of precision optical instrument repair, cultural relic restoration, and identification of other precious device broken accidents. Different from general intact transparent objects, the contour estimation of transparent fragments face greater challenges due to strict optical properties, irregular shapes and edges. To address this issue, a general transparent fragments contour estimation framework based on visual-tactile fusion is proposed in this paper. First, we construct the transparent fragment dataset named TransFrag27K, which includes a multiscene synthetic data of broken fragments from multiple types of transparent objects, and a scalable synthetic data generation pipeline. Secondly, we propose a visual grasping position detection network named TransFragNet to identify, locate and segment the sampling grasping position. And, we use a two-finger gripper with Gelsight Mini sensors to obtain reconstructed tactile information of the lateral edge of the fragments. By fusing this tactile information with visual cues, a visual-tactile fusion material classifier is proposed. Inspired by the way humans estimate a fragment's contour combining vision and touch, we introduce a general transparent fragment contour estimation framework based on visual-tactile fusion, demonstrates strong performance in real-world validation. Finally, a multi-dimensional similarity metrics based contour matching and reassembly algorithm is proposed, providing a reproducible benchmark for evaluating visual-tactile contour estimation and fragment reassembly. The experimental results demonstrate the validity of the proposed framework. The dataset and codes are available at https://github.com/Keithllin/Transparent-Fragments-Contour-Estimation.