# Trans4Trans: Efficient Transformer for Transparent Object Segmentation to Help Visually Impaired People Navigate in the Real World

arXiv: https://arxiv.org/abs/2107.03172 · HTML: https://mlsanity.com/paper/2107.03172/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 85 references that are in the map.
- First posted on arXiv 7 Jul 2021.

## Builds on (most cited first, 30 of 51)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Dual Attention Network for Scene Segmentation](https://mlsanity.com/paper/1809.02983/) (2018)
- [Segmenter: Transformer for Semantic Segmentation](https://mlsanity.com/paper/2105.05633/) (2021)
- [CCNet: Criss-Cross Attention for Semantic Segmentation](https://mlsanity.com/paper/1811.11721/) (2018)
- [BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation](https://mlsanity.com/paper/1808.00897/) (2018)
- [LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference](https://mlsanity.com/paper/2104.01136/) (2021)
- [RefineNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentation](https://mlsanity.com/paper/1611.06612/) (2016)
- [Joint 2D-3D-Semantic Data for Indoor Scene Understanding](https://mlsanity.com/paper/1702.01105/) (2017)
- [MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers](https://mlsanity.com/paper/2012.00759/) (2020)
- [ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation](https://mlsanity.com/paper/1606.02147/) (2016)
- [ICNet for Real-Time Semantic Segmentation on High-Resolution Images](https://mlsanity.com/paper/1704.08545/) (2017)
- [Asymmetric Non-local Neural Networks for Semantic Segmentation](https://mlsanity.com/paper/1908.07678/) (2019)

## Built on by (most cited first, 30 of 30)

- [CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformers](https://mlsanity.com/paper/2203.04838/) (2022)
- [MatchFormer: Interleaving Attention in Transformers for Feature Matching](https://mlsanity.com/paper/2203.09645/) (2022)
- [Bending Reality: Distortion-aware Transformers for Adapting to Panoramic Semantic Segmentation](https://mlsanity.com/paper/2203.01452/) (2022)
- [Behind Every Domain There is a Shift: Adapting Distortion-aware Vision Transformers for Panoramic Semantic Segmentation](https://mlsanity.com/paper/2207.11860/) (2022)
- [Transfer beyond the Field of View: Dense Panoramic Semantic Segmentation via Unsupervised Domain Adaptation](https://mlsanity.com/paper/2110.11062/) (2021)
- [Glass Segmentation with RGB-Thermal Image Pairs](https://mlsanity.com/paper/2204.05453/) (2022)
- [Trans4Trans: Efficient Transformer for Transparent Object and Semantic Scene Segmentation in Real-World Navigation Assistance](https://mlsanity.com/paper/2108.09174/) (2021)
- [360BEV: Panoramic Semantic Mapping for Indoor Bird's-Eye View](https://mlsanity.com/paper/2303.11910/) (2023)
- [MVTrans: Multi-View Perception of Transparent Objects](https://mlsanity.com/paper/2302.11683/) (2023)
- [Delving Deep into One-Shot Skeleton-based Action Recognition with Diverse Occlusions](https://mlsanity.com/paper/2202.11423/) (2022)
- [Open Scene Understanding: Grounded Situation Recognition Meets Segment Anything for Helping People with Visual Impairments](https://mlsanity.com/paper/2307.07757/) (2023)
- [Continual BatchNorm Adaptation (CBNA) for Semantic Segmentation](https://mlsanity.com/paper/2203.01074/) (2022)
- [MateRobot: Material Recognition in Wearable Robotics for People with Visual Impairments](https://mlsanity.com/paper/2302.14595/) (2023)
- [Efficiency 360: Efficient Vision Transformers](https://mlsanity.com/paper/2302.08374/) (2023)
- [Flying Guide Dog: Walkable Path Discovery for the Visually Impaired Utilizing Drones and Transformer-based Semantic Segmentation](https://mlsanity.com/paper/2108.07007/) (2021)
- [Indoor Navigation Assistance for Visually Impaired People via Dynamic SLAM and Panoptic Segmentation with an RGB-D Sensor](https://mlsanity.com/paper/2204.01154/) (2022)
- [ObjectFinder: An Open-Vocabulary Assistive System for Interactive Object Search by Blind People](https://mlsanity.com/paper/2412.03118/) (2024)
- [Panoramic Depth Estimation via Supervised and Unsupervised Learning in Indoor Scenes](https://mlsanity.com/paper/2108.08076/) (2021)
- [Predicting 3D shapes, masks, and properties of materials, liquids, and objects inside transparent containers, using the TransProteus CGI dataset](https://mlsanity.com/paper/2109.07577/) (2021)
- [On the Choice of Data for Efficient Training and Validation of End-to-End Driving Models](https://mlsanity.com/paper/2206.00608/) (2022)
- [@Bench: Benchmarking Vision-Language Models for Human-centered Assistive Technology](https://mlsanity.com/paper/2409.14215/) (2024)
- [SeeClear: Reliable Transparent Object Depth Estimation via Generative Opacification](https://mlsanity.com/paper/2603.19547/) (2026)
- [Influence of Water Droplet Contamination for Transparency Segmentation](https://mlsanity.com/paper/2405.12861/) (2024)
- [Comb, Prune, Distill: Towards Unified Pruning for Vision Model Compression](https://mlsanity.com/paper/2408.03046/) (2024)
- [Automating Catheterization Labs with Real-Time Perception](https://mlsanity.com/paper/2403.05758/) (2024)
- [Semantic Segmentation of Transparent and Opaque Drinking Glasses with the Help of Zero-shot Learning](https://mlsanity.com/paper/2503.15004/) (2025)
- [Glass Surface Detection: Leveraging Reflection Dynamics in Flash/No-flash Imagery](https://mlsanity.com/paper/2511.16887/) (2025)
- [SEP-YOLO: Fourier-Domain Feature Representation for Transparent Object Instance Segmentation](https://mlsanity.com/paper/2603.02648/) (2026)
- [Part-Aware Open-Vocabulary 3D Affordance Grounding via Prototypical Semantic and Geometric Alignment](https://mlsanity.com/paper/2603.17647/) (2026)
- [GlassFormer: Learning Real-time Glass Segmentation using Radar-Depth Fusion](https://mlsanity.com/paper/2609.36844/) (2026)

## Abstract (arXiv)

Common fully glazed facades and transparent objects present architectural barriers and impede the mobility of people with low vision or blindness, for instance, a path detected behind a glass door is inaccessible unless it is correctly perceived and reacted. However, segmenting these safety-critical objects is rarely covered by conventional assistive technologies. To tackle this issue, we construct a wearable system with a novel dual-head Transformer for Transparency (Trans4Trans) model, which is capable of segmenting general and transparent objects and performing real-time wayfinding to assist people walking alone more safely. Especially, both decoders created by our proposed Transformer Parsing Module (TPM) enable effective joint learning from different datasets. Besides, the efficient Trans4Trans model composed of symmetric transformer-based encoder and decoder, requires little computational expenses and is readily deployed on portable GPUs. Our Trans4Trans model outperforms state-of-the-art methods on the test sets of Stanford2D3D and Trans10K-v2 datasets and obtains mIoU of 45.13% and 75.14%, respectively. Through various pre-tests and a user study conducted in indoor and outdoor scenarios, the usability and reliability of our assistive system have been extensively verified.