# DFormerv2: Geometry Self-Attention for RGBD Semantic Segmentation

arXiv: https://arxiv.org/abs/2504.04701 · HTML: https://mlsanity.com/paper/2504.04701/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 57 of its 74 references that are in the map.
- First posted on arXiv 7 Apr 2025.

## Builds on (most cited first, 30 of 57)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021)
- [Segmenter: Transformer for Semantic Segmentation](https://mlsanity.com/paper/2105.05633/) (2021)
- [PVT v2: Improved Baselines with Pyramid Vision Transformer](https://mlsanity.com/paper/2106.13797/) (2021)
- [Retentive Network: A Successor to Transformer for Large Language Models](https://mlsanity.com/paper/2307.08621/) (2023)
- [CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows](https://mlsanity.com/paper/2107.00652/) (2021)
- [CMT: Convolutional Neural Networks Meet Vision Transformers](https://mlsanity.com/paper/2107.06263/) (2021)
- [Vision Transformer with Deformable Attention](https://mlsanity.com/paper/2201.00520/) (2022)
- [MultiMAE: Multi-modal Multi-task Masked Autoencoders](https://mlsanity.com/paper/2204.01678/) (2022)
- [Focal Self-attention for Local-Global Interactions in Vision Transformers](https://mlsanity.com/paper/2107.00641/) (2021)
- [SegNeXt: Rethinking Convolutional Attention Design for Semantic Segmentation](https://mlsanity.com/paper/2209.08575/) (2022)
- [Neighborhood Attention Transformer](https://mlsanity.com/paper/2204.07143/) (2022)
- [CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformers](https://mlsanity.com/paper/2203.04838/) (2022)
- [Omnivore: A Single Model for Many Visual Modalities](https://mlsanity.com/paper/2201.08377/) (2022)
- [A Simple Pooling-Based Design for Real-Time Salient Object Detection](https://mlsanity.com/paper/1904.09569/) (2019)
- [Co-SLAM: Joint Coordinate and Sparse Parametric Encodings for Neural Real-Time SLAM](https://mlsanity.com/paper/2304.14377/) (2023)
- [BiFormer: Vision Transformer with Bi-Level Routing Attention](https://mlsanity.com/paper/2303.08810/) (2023)
- [Pattern-Affinitive Propagation across Depth, Surface Normal and Semantic Segmentation](https://mlsanity.com/paper/1906.03525/) (2019)
- [CrossFormer: A Versatile Vision Transformer Hinging on Cross-scale Attention](https://mlsanity.com/paper/2108.00154/) (2021)
- [Multimodal Token Fusion for Vision Transformers](https://mlsanity.com/paper/2204.08721/) (2022)
- [Bi-directional Cross-Modality Feature Propagation with Separation-and-Aggregation Gate for RGB-D Semantic Segmentation](https://mlsanity.com/paper/2007.09183/) (2020)

## Built on by (most cited first, 30 of 31)

- [Depth Anything at Any Condition](https://mlsanity.com/paper/2507.01634/) (2025)
- [Revisiting Efficient Semantic Segmentation: Learning Offsets for Better Spatial and Class Feature Alignment](https://mlsanity.com/paper/2508.08811/) (2025)
- [OmniSegmentor: A Flexible Multi-Modal Learning Framework for Semantic Segmentation](https://mlsanity.com/paper/2509.15096/) (2025)
- [Unleashing Semantic and Geometric Priors for 3D Scene Completion](https://mlsanity.com/paper/2508.13601/) (2025)
- [TUNI: Unifying Pre-training and Fine-tuning with Modality-Aware Mutual Learning and Rectification for RGB-T Semantic Segmentation](https://mlsanity.com/paper/2509.10005/) (2025)
- [DiffPixelFormer: Differential Pixel-Aware Transformer for RGB-D Indoor Scene Segmentation](https://mlsanity.com/paper/2511.13047/) (2025)
- [Crowded Video Individual Counting Informed by Social Grouping and Spatial-Temporal Displacement Priors](https://mlsanity.com/paper/2601.01192/) (2026)
- [RA-SOD: Reliability-Aware RGB-T Salient Object Detection under Modality Degradation](https://mlsanity.com/paper/2609.12622/) (2026)
- [Geo-RepNet: Geometry-Aware Representation Learning for Surgical Phase Recognition in Endoscopic Submucosal Dissection](https://mlsanity.com/paper/2507.09294/) (2025)
- [GeomPrompt: Geometric Prompt Learning for RGB-D Semantic Segmentation Under Missing and Degraded Depth](https://mlsanity.com/paper/2604.11585/) (2026)
- [RSGMamba: Reliability-Aware Self-Gated State Space Model for Multimodal Semantic Segmentation](https://mlsanity.com/paper/2604.12319/) (2026)
- [Robust Multispectral Semantic Segmentation under Missing or Full Modalities via Structured Latent Projection](https://mlsanity.com/paper/2604.15856/) (2026)
- [Structure-Semantic Decoupled Modulation of Global Geospatial Embeddings for High-Resolution Remote Sensing Mapping](https://mlsanity.com/paper/2604.19591/) (2026)
- [RelFlexformer: Efficient Attention 3D-Transformers for Integrable Relative Positional Encodings](https://mlsanity.com/paper/2605.10706/) (2026)
- [Geometry-aware Gaussian Prior and Axial Attention for Cervical Cytology Image Classification](https://mlsanity.com/paper/2607.10278/) (2026)
- [DGA-Net: Enhancing SAM with Depth Prompting and Graph-Anchor Guidance for Camouflaged Object Detection](https://mlsanity.com/paper/2601.02831/) (2026)
- [Efficient Segment Anything with Depth-Aware Fusion and Limited Training Data](https://mlsanity.com/paper/2602.11804/) (2026)
- [Two-Stream Interactive Joint Learning of Scene Parsing and Geometric Vision Tasks](https://mlsanity.com/paper/2602.13588/) (2026)
- [RoboPCA: Pose-centered Affordance Learning from Human Demonstrations for Robot Manipulation](https://mlsanity.com/paper/2603.07691/) (2026)
- [DriveXQA: Cross-modal Visual Question Answering for Adverse Driving Scene Understanding](https://mlsanity.com/paper/2603.11380/) (2026)
- [Unified Restoration-Perception Learning: Maritime Infrared-Visible Image Fusion and Segmentation](https://mlsanity.com/paper/2603.28414/) (2026)
- [Graph-based Semantic Calibration Network for Unaligned UAV RGBT Image Semantic Segmentation and A Large-scale Benchmark](https://mlsanity.com/paper/2604.26893/) (2026)
- [BED-SAM2: Boundary-Enhanced-Depth SAM2 via Monocular Geometric Priors](https://mlsanity.com/paper/2605.24893/) (2026)
- [AFUN: Towards an Affordance Foundation Model for Functionality Understanding](https://mlsanity.com/paper/2606.02551/) (2026)
- [Weaving Light and Time: Unified Harmonic-Geometric Representation Learning for Dense RGB-Event Parsing](https://mlsanity.com/paper/2607.09143/) (2026)
- [Compass: Degradation-Simulated Reciprocal Learning with Lightweight Needle RWKV for Multimodal Crack Segmentation under Missing Modalities](https://mlsanity.com/paper/2608.03559/) (2026)
- [URNet: A Unified Reparameterized Network for Efficient RGB-D Semantic Segmentation](https://mlsanity.com/paper/2608.05671/) (2026)
- [DINOcular: Self-Supervised Visuospatial Representations](https://mlsanity.com/paper/2608.27226/) (2026)
- [Privacy-Preserving Semantic Segmentation from High-Resolution Depth and Ultra-Low-Resolution RGB](https://mlsanity.com/paper/2609.28360/) (2026)
- [RGBD20K: A Large-Scale Benchmark for RGB-D Semantic Segmentation](https://mlsanity.com/paper/2609.29028/) (2026)

## Abstract (arXiv)

Recent advances in scene understanding benefit a lot from depth maps because of the 3D geometry information, especially in complex conditions (e.g., low light and overexposed). Existing approaches encode depth maps along with RGB images and perform feature fusion between them to enable more robust predictions. Taking into account that depth can be regarded as a geometry supplement for RGB images, a straightforward question arises: Do we really need to explicitly encode depth information with neural networks as done for RGB images? Based on this insight, in this paper, we investigate a new way to learn RGBD feature representations and present DFormerv2, a strong RGBD encoder that explicitly uses depth maps as geometry priors rather than encoding depth information with neural networks. Our goal is to extract the geometry clues from the depth and spatial distances among all the image patch tokens, which will then be used as geometry priors to allocate attention weights in self-attention. Extensive experiments demonstrate that DFormerv2 exhibits exceptional performance in various RGBD semantic segmentation benchmarks. Code is available at: https://github.com/VCIP-RGBD/DFormer.