# Transparent Object Tracking with Enhanced Fusion Module

arXiv: https://arxiv.org/abs/2309.06701 · HTML: https://mlsanity.com/paper/2309.06701/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 35 references that are in the map.
- First posted on arXiv 13 Sep 2023.

## Builds on (most cited first, 24 of 24)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wild](https://mlsanity.com/paper/1810.11981/) (2018)
- [LaSOT: A High-quality Benchmark for Large-scale Single Object Tracking](https://mlsanity.com/paper/1809.07845/) (2018)
- [Transformer Tracking](https://mlsanity.com/paper/2103.15436/) (2021)
- [Learning Spatio-Temporal Transformer for Visual Tracking](https://mlsanity.com/paper/2103.17154/) (2021)
- [Learning Discriminative Model Prediction for Tracking](https://mlsanity.com/paper/1904.07220/) (2019)
- [TrackingNet: A Large-Scale Dataset and Benchmark for Object Tracking in the Wild](https://mlsanity.com/paper/1803.10794/) (2018)
- [ATOM: Accurate Tracking by Overlap Maximization](https://mlsanity.com/paper/1811.07628/) (2018)
- [AiATrack: Attention in Attention for Transformer Visual Tracking](https://mlsanity.com/paper/2207.09603/) (2022)
- [Learning Target Candidate Association to Keep Track of What Not to Track](https://mlsanity.com/paper/2103.16556/) (2021)
- [ClearGrasp: 3D Shape Estimation of Transparent Objects for Manipulation](https://mlsanity.com/paper/1910.02550/) (2019)
- [Transforming Model Prediction for Tracking](https://mlsanity.com/paper/2203.11192/) (2022)
- [Segmenting Transparent Objects in the Wild](https://mlsanity.com/paper/2003.13948/) (2020)
- [Real-time Multiple People Tracking with Deeply Learned Candidate Selection and Person Re-Identification](https://mlsanity.com/paper/1809.04427/) (2018)
- [Everything at Once -- Multi-modal Fusion Transformer for Video Retrieval](https://mlsanity.com/paper/2112.04446/) (2021)
- [Jointly Modeling Motion and Appearance Cues for Robust RGB-T Tracking](https://mlsanity.com/paper/2007.02041/) (2020)
- [Robust Visual Tracking by Segmentation](https://mlsanity.com/paper/2203.11191/) (2022)
- [TOM-Net: Learning Transparent Object Matting from a Single Image](https://mlsanity.com/paper/1803.04636/) (2018)
- [TransCut: Transparent Object Segmentation from a Light-Field Image](https://mlsanity.com/paper/1511.06853/) (2015)
- [Transparent Object Tracking Benchmark](https://mlsanity.com/paper/2011.10875/) (2020)
- [Multi-modal Transfer Learning for Grasping Transparent and Specular Objects](https://mlsanity.com/paper/2006.00028/) (2020)
- [Depth Masked Discriminative Correlation Filter](https://mlsanity.com/paper/1802.09227/) (2018)
- [Trans2k: Unlocking the Power of Deep Models for Transparent Object Tracking](https://mlsanity.com/paper/2210.03436/) (2022)

## Built on by (most cited first, 1 of 1)

- [ClearDepth: Enhanced Stereo Perception of Transparent Objects for Robotic Manipulation](https://mlsanity.com/paper/2409.08926/) (2024)

## Abstract (arXiv)

Accurate tracking of transparent objects, such as glasses, plays a critical role in many robotic tasks such as robot-assisted living. Due to the adaptive and often reflective texture of such objects, traditional tracking algorithms that rely on general-purpose learned features suffer from reduced performance. Recent research has proposed to instill transparency awareness into existing general object trackers by fusing purpose-built features. However, with the existing fusion techniques, the addition of new features causes a change in the latent space making it impossible to incorporate transparency awareness on trackers with fixed latent spaces. For example, many of the current days transformer-based trackers are fully pre-trained and are sensitive to any latent space perturbations. In this paper, we present a new feature fusion technique that integrates transparency information into a fixed feature space, enabling its use in a broader range of trackers. Our proposed fusion module, composed of a transformer encoder and an MLP module, leverages key query-based transformations to embed the transparency information into the tracking pipeline. We also present a new two-step training strategy for our fusion module to effectively merge transparency features. We propose a new tracker architecture that uses our fusion techniques to achieve superior results for transparent object tracking. Our proposed method achieves competitive results with state-of-the-art trackers on TOTB, which is the largest transparent object tracking benchmark recently released. Our results and the implementation of code will be made publicly available at https://github.com/kalyan0510/TOTEM.