# Improving Depth Gradient Continuity in Transformers: A Comparative Study on Monocular Depth Estimation with CNN

arXiv: https://arxiv.org/abs/2308.08333 · HTML: https://mlsanity.com/paper/2308.08333/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 37 references that are in the map.
- First posted on arXiv 16 Aug 2023.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [Quantifying Attention Flow in Transformers](https://mlsanity.com/paper/2005.00928/) (2020)
- [Deep Ordinal Regression Network for Monocular Depth Estimation](https://mlsanity.com/paper/1806.02446/) (2018)
- [Transformer Interpretability Beyond Attention Visualization](https://mlsanity.com/paper/2012.09838/) (2020)
- [AdaBins: Depth Estimation using Adaptive Bins](https://mlsanity.com/paper/2011.14141/) (2020)
- [Escaping the Big Data Paradigm with Compact Transformers](https://mlsanity.com/paper/2104.05704/) (2021)
- [Enforcing geometric constraints of virtual normal for depth prediction](https://mlsanity.com/paper/1907.12209/) (2019)
- [Revisiting Stereo Depth Estimation From a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2011.02910/) (2020)
- [NeW CRFs: Neural Window Fully-connected CRFs for Monocular Depth Estimation](https://mlsanity.com/paper/2203.01502/) (2022)
- [Revisiting Single Image Depth Estimation: Toward Higher Resolution Maps with Accurate Object Boundaries](https://mlsanity.com/paper/1803.08673/) (2018)
- [Transformer-Based Attention Networks for Continuous Pixel-Wise Prediction](https://mlsanity.com/paper/2103.12091/) (2021)
- [Lite-Mono: A Lightweight CNN and Transformer Architecture for Self-Supervised Monocular Depth Estimation](https://mlsanity.com/paper/2211.13202/) (2022)
- [Attention Attention Everywhere: Monocular Depth Prediction with Skip Attention](https://mlsanity.com/paper/2210.09071/) (2022)
- [Global-Local Path Networks for Monocular Depth Estimation with Vertical CutDepth](https://mlsanity.com/paper/2201.07436/) (2022)
- [CNN Explainer: Learning Convolutional Neural Networks with Interactive Visualization](https://mlsanity.com/paper/2004.15004/) (2020)
- [Visualization of Convolutional Neural Networks for Monocular Depth Estimation](https://mlsanity.com/paper/1904.03380/) (2019)
- [Depthformer : Multiscale Vision Transformer For Monocular Depth Estimation With Local Global Information Fusion](https://mlsanity.com/paper/2207.04535/) (2022)
- [SwinDepth: Unsupervised Depth Estimation using Monocular Sequences via Swin Transformer and Densely Cascaded Network](https://mlsanity.com/paper/2301.06715/) (2023)
- [DEHRFormer: Real-time Transformer for Depth Estimation and Haze Removal from Varicolored Haze Scenes](https://mlsanity.com/paper/2303.06905/) (2023)
- [DwinFormer: Dual Window Transformers for End-to-End Monocular Depth Estimation](https://mlsanity.com/paper/2303.02968/) (2023)

## Built on by (most cited first, 30 of 35)

- [Retinexmamba: Retinex-based Mamba for Low-light Image Enhancement](https://mlsanity.com/paper/2405.03349/) (2024)
- [Deception Detection from Linguistic and Physiological Data Streams Using Bimodal Convolutional Neural Networks](https://mlsanity.com/paper/2311.10944/) (2023)
- [Exploring Diverse Methods in Visual Question Answering](https://mlsanity.com/paper/2404.13565/) (2024)
- [Self-Supervised Graph Neural Networks for Enhanced Feature Extraction in Heterogeneous Information Networks](https://mlsanity.com/paper/2410.17617/) (2024)
- [Residual-based Language Models are Free Boosters for Biomedical Imaging](https://mlsanity.com/paper/2403.17343/) (2024)
- [A Comparative Study on Enhancing Prediction in Social Network Advertisement through Data Augmentation](https://mlsanity.com/paper/2404.13812/) (2024)
- [Utilizing Deep Learning to Optimize Software Development Processes](https://mlsanity.com/paper/2404.13630/) (2024)
- [Efficiency optimization of large-scale language models based on deep learning in natural language processing tasks](https://mlsanity.com/paper/2405.11704/) (2024)
- [Building Lane-Level Maps from Aerial Images](https://mlsanity.com/paper/2312.13449/) (2023)
- [Application of Multimodal Fusion Deep Learning Model in Disease Recognition](https://mlsanity.com/paper/2406.18546/) (2024)
- [A comparative study of generative adversarial networks for image recognition algorithms based on deep learning and traditional methods](https://mlsanity.com/paper/2408.03568/) (2024)
- [Research on Image Recognition Technology Based on Multimodal Deep Learning](https://mlsanity.com/paper/2405.03091/) (2024)
- [Exploration of Attention Mechanism-Enhanced Deep Learning Models in the Mining of Medical Textual Data](https://mlsanity.com/paper/2406.00016/) (2024)
- [Enhancing Medical Imaging with GANs Synthesizing Realistic Images from Limited Data](https://mlsanity.com/paper/2406.18547/) (2024)
- [Advanced Financial Fraud Detection Using GNN-CL Model](https://mlsanity.com/paper/2407.06529/) (2024)
- [Optimizing YOLOv5s Object Detection through Knowledge Distillation algorithm](https://mlsanity.com/paper/2410.12259/) (2024)
- [Top-K Pooling with Patch Contrastive Learning for Weakly-Supervised Semantic Segmentation](https://mlsanity.com/paper/2310.09828/) (2023)
- [From Handcrafted Features to LLMs: A Brief Survey for Machine Translation Quality Estimation](https://mlsanity.com/paper/2403.14118/) (2024)
- [Exploration of Multi-Scale Image Fusion Systems in Intelligent Medical Image Analysis](https://mlsanity.com/paper/2406.18548/) (2024)
- [Algorithm Research of ELMo Word Embedding and Deep Learning Multimodal Transformer in Image Description](https://mlsanity.com/paper/2408.06357/) (2024)
- [UniHead: Unifying Multi-Perception for Detection Heads](https://mlsanity.com/paper/2309.13242/) (2023)
- [Research on Intelligent Aided Diagnosis System of Medical Image Based on Computer Deep Learning](https://mlsanity.com/paper/2404.18419/) (2024)
- [Integrating Medical Imaging and Clinical Reports Using Multimodal Deep Learning for Advanced Disease Analysis](https://mlsanity.com/paper/2405.17459/) (2024)
- [Robust Graph Neural Networks for Stability Analysis in Dynamic Networks](https://mlsanity.com/paper/2411.11848/) (2024)
- [Application of Computer Deep Learning Model in Diagnosis of Pulmonary Nodules](https://mlsanity.com/paper/2406.13205/) (2024)
- [Joint-Motion Mutual Learning for Pose Estimation in Videos](https://mlsanity.com/paper/2408.02285/) (2024)
- [Adaptive parameter sharing for multi-agent reinforcement learning](https://mlsanity.com/paper/2312.09009/) (2023)
- [Research on Early Warning Model of Cardiovascular Disease Based on Computer Deep Learning](https://mlsanity.com/paper/2406.08864/) (2024)
- [Research on Disease Prediction Model Construction Based on Computer AI deep Learning Technology](https://mlsanity.com/paper/2406.16982/) (2024)
- [Investigation of Customized Medical Decision Algorithms Utilizing Graph Neural Networks](https://mlsanity.com/paper/2405.17460/) (2024)

## Abstract (arXiv)

Monocular depth estimation is an ongoing challenge in computer vision. Recent progress with Transformer models has demonstrated notable advantages over conventional CNNs in this area. However, there's still a gap in understanding how these models prioritize different regions in 2D images and how these regions affect depth estimation performance. To explore the differences between Transformers and CNNs, we employ a sparse pixel approach to contrastively analyze the distinctions between the two. Our findings suggest that while Transformers excel in handling global context and intricate textures, they lag behind CNNs in preserving depth gradient continuity. To further enhance the performance of Transformer models in monocular depth estimation, we propose the Depth Gradient Refinement (DGR) module that refines depth estimation through high-order differentiation, feature fusion, and recalibration. Additionally, we leverage optimal transport theory, treating depth maps as spatial probability distributions, and employ the optimal transport distance as a loss function to optimize our model. Experimental results demonstrate that models integrated with the plug-and-play Depth Gradient Refinement (DGR) module and the proposed loss function enhance performance without increasing complexity and computational costs on both outdoor KITTI and indoor NYU-Depth-v2 datasets. This research not only offers fresh insights into the distinctions between Transformers and CNNs in depth estimation but also paves the way for novel depth estimation methodologies.