# Glass Segmentation with RGB-Thermal Image Pairs

arXiv: https://arxiv.org/abs/2204.05453 · HTML: https://mlsanity.com/paper/2204.05453/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 59 of its 132 references that are in the map.
- First posted on arXiv 12 Apr 2022.

## Builds on (most cited first, 30 of 59)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Segmenter: Transformer for Semantic Segmentation](https://mlsanity.com/paper/2105.05633/) (2021)
- [Image Transformer](https://mlsanity.com/paper/1802.05751/) (2018)
- [Open3D: A Modern Library for 3D Data Processing](https://mlsanity.com/paper/1801.09847/) (2018)
- [AdaBins: Depth Estimation using Adaptive Bins](https://mlsanity.com/paper/2011.14141/) (2020)
- [Structure-measure: A New Way to Evaluate Foreground Maps](https://mlsanity.com/paper/1708.00786/) (2017)
- [Multi-Modal Fusion Transformer for End-to-End Autonomous Driving](https://mlsanity.com/paper/2104.09224/) (2021)
- [CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformers](https://mlsanity.com/paper/2203.04838/) (2022)
- [Multi-scale Interactive Network for Salient Object Detection](https://mlsanity.com/paper/2007.09062/) (2020)
- [CenterMask : Real-Time Anchor-Free Instance Segmentation](https://mlsanity.com/paper/1911.06667/) (2019)
- [The Secrets of Salient Object Segmentation](https://mlsanity.com/paper/1406.2807/) (2014)
- [PLOP: Learning without Forgetting for Continual Semantic Segmentation](https://mlsanity.com/paper/2011.11390/) (2020)
- [Visual Saliency Transformer](https://mlsanity.com/paper/2104.12099/) (2021)
- [ClearGrasp: 3D Shape Estimation of Transparent Objects for Manipulation](https://mlsanity.com/paper/1910.02550/) (2019)
- [RGB-T Object Tracking:Benchmark and Baseline](https://mlsanity.com/paper/1805.08982/) (2018)
- [Segmenting Transparent Objects in the Wild](https://mlsanity.com/paper/2003.13948/) (2020)
- [UC-Net: Uncertainty Inspired RGB-D Saliency Detection via Conditional Variational Autoencoders](https://mlsanity.com/paper/2004.05763/) (2020)
- [PST900: RGB-Thermal Calibration, Dataset and Segmentation Network](https://mlsanity.com/paper/1909.10980/) (2019)
- [Efficient RGB-D Semantic Segmentation for Indoor Scene Analysis](https://mlsanity.com/paper/2011.06961/) (2020)
- [JL-DCF: Joint Learning and Densely-Cooperative Fusion Framework for RGB-D Salient Object Detection](https://mlsanity.com/paper/2004.08515/) (2020)
- [ShapeConv: Shape-aware Convolutional Layer for Indoor RGB-D Semantic Segmentation](https://mlsanity.com/paper/2108.10528/) (2021)
- [Line Segment Detection Using Transformers without Edges](https://mlsanity.com/paper/2101.01909/) (2021)
- [Hierarchical Dynamic Filtering Network for RGB-D Salient Object Detection](https://mlsanity.com/paper/2007.06227/) (2020)
- [Specificity-preserving RGB-D Saliency Detection](https://mlsanity.com/paper/2108.08162/) (2021)

## Built on by (most cited first, 26 of 26)

- [RGB-T Semantic Segmentation with Location, Activation, and Sharpening](https://mlsanity.com/paper/2210.14530/) (2022)
- [RFENet: Towards Reciprocal Feature Evolution for Glass Segmentation](https://mlsanity.com/paper/2307.06099/) (2023)
- [TRansPose: Large-Scale Multispectral Dataset for Transparent Object](https://mlsanity.com/paper/2307.05016/) (2023)
- [Exploring Multi-modal Neural Scene Representations With Applications on Thermal Imaging](https://mlsanity.com/paper/2403.11865/) (2024)
- [Object Segmentation by Mining Cross-Modal Semantics](https://mlsanity.com/paper/2305.10469/) (2023)
- [TranSplat: Surface Embedding-guided 3D Gaussian Splatting for Transparent Object Manipulation](https://mlsanity.com/paper/2502.07840/) (2025)
- [What Happened 3 Seconds Ago? Inferring the Past with Thermal Imaging](https://mlsanity.com/paper/2304.13651/) (2023)
- [A Multispectral Automated Transfer Technique (MATT) for machine-driven image labeling utilizing the Segment Anything Model (SAM)](https://mlsanity.com/paper/2402.11413/) (2024)
- [Monocular Depth Estimation and Segmentation for Transparent Object with Iterative Semantic and Geometric Fusion](https://mlsanity.com/paper/2502.14616/) (2025)
- [M-SpecGene: Generalized Foundation Model for RGBT Multispectral Vision](https://mlsanity.com/paper/2507.16318/) (2025)
- [Deep Learning in Concealed Dense Prediction](https://mlsanity.com/paper/2504.10979/) (2025)
- [GEM: Boost Simple Network for Glass Surface Segmentation via Vision Foundation Models](https://mlsanity.com/paper/2307.12018/) (2023)
- [Glass Surface Segmentation with an RGB-D Camera via Weighted Feature Fusion for Service Robots](https://mlsanity.com/paper/2508.01639/) (2025)
- [Don't worry about mistakes! Glass Segmentation Network via Mistake Correction](https://mlsanity.com/paper/2304.10825/) (2023)
- [Influence of Water Droplet Contamination for Transparency Segmentation](https://mlsanity.com/paper/2405.12861/) (2024)
- [Shaken, Not Stirred: A Novel Dataset for Visual Understanding of Glasses in Human-Robot Bartending Tasks](https://mlsanity.com/paper/2503.04308/) (2025)
- [SimMAT: Exploring Transferability from Vision Foundation Models to Any Image Modality](https://mlsanity.com/paper/2409.08083/) (2024)
- [Indoor Heat Estimation from a Single Visible-Light Panorama](https://mlsanity.com/paper/2502.06973/) (2025)
- [Glass Surface Detection: Leveraging Reflection Dynamics in Flash/No-flash Imagery](https://mlsanity.com/paper/2511.16887/) (2025)
- [MVGD-Net: A Novel Motion-aware Video Glass Surface Detection Method](https://mlsanity.com/paper/2601.13715/) (2026)
- [SEP-YOLO: Fourier-Domain Feature Representation for Transparent Object Instance Segmentation](https://mlsanity.com/paper/2603.02648/) (2026)
- [Glass Segmentation with Fusion of Learned and General Visual Features](https://mlsanity.com/paper/2603.03718/) (2026)
- [Transparent Fragments Contour Estimation via Visual-Tactile Fusion for Autonomous Reassembly](https://mlsanity.com/paper/2603.20290/) (2026)
- [Enhancing Glass Surface Reconstruction via Depth Prior for Robot Navigation](https://mlsanity.com/paper/2604.18336/) (2026)
- [Glass Surface Detection Grounded in 3D Visual Geometry](https://mlsanity.com/paper/2608.26752/) (2026)
- [GlassFormer: Learning Real-time Glass Segmentation using Radar-Depth Fusion](https://mlsanity.com/paper/2609.36844/) (2026)

## Abstract (arXiv)

This paper proposes a new glass segmentation method utilizing paired RGB and thermal images. Due to the large difference between the transmission property of visible light and that of the thermal energy through the glass where most glass is transparent to the visible light but opaque to thermal energy, glass regions of a scene are made more distinguishable with a pair of RGB and thermal images than solely with an RGB image. To exploit such a unique property, we propose a neural network architecture that effectively combines an RGB-thermal image pair with a new multi-modal fusion module based on attention, and integrate CNN and transformer to extract local features and non-local dependencies, respectively. As well, we have collected a new dataset containing 5551 RGB-thermal image pairs with ground-truth segmentation annotations. The qualitative and quantitative evaluations demonstrate the effectiveness of the proposed approach on fusing RGB and thermal data for glass segmentation. Our code and data are available at https://github.com/Dong-Huo/RGB-T-Glass-Segmentation.