# Depth Estimation Algorithm Based on Transformer-Encoder and Feature Fusion

arXiv: https://arxiv.org/abs/2403.01370 · HTML: https://mlsanity.com/paper/2403.01370/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 23 references that are in the map.
- First posted on arXiv 3 Mar 2024.

## Builds on (most cited first, 16 of 16)

- [AdaBins: Depth Estimation using Adaptive Bins](https://mlsanity.com/paper/2011.14141/) (2020)
- [Unifying Flow, Stereo and Depth Estimation](https://mlsanity.com/paper/2211.05783/) (2022)
- [Enforcing geometric constraints of virtual normal for depth prediction](https://mlsanity.com/paper/1907.12209/) (2019)
- [Unsupervised Learning of Monocular Depth Estimation and Visual Odometry with Deep Feature Reconstruction](https://mlsanity.com/paper/1803.03893/) (2018)
- [NeW CRFs: Neural Window Fully-connected CRFs for Monocular Depth Estimation](https://mlsanity.com/paper/2203.01502/) (2022)
- [Pattern-Affinitive Propagation across Depth, Surface Normal and Semantic Segmentation](https://mlsanity.com/paper/1906.03525/) (2019)
- [DepthFormer: Exploiting Long-Range Correlation and Local Information for Accurate Monocular Depth Estimation](https://mlsanity.com/paper/2203.14211/) (2022)
- [Self-supervised Monocular Trained Depth Estimation using Self-attention and Discrete Disparity Volume](https://mlsanity.com/paper/2003.13951/) (2020)
- [Multi-Scale Continuous CRFs as Sequential Deep Networks for Monocular Depth Estimation](https://mlsanity.com/paper/1704.02157/) (2017)
- [Single Image Depth Estimation: An Overview](https://mlsanity.com/paper/2104.06456/) (2021)
- [Depthformer : Multiscale Vision Transformer For Monocular Depth Estimation With Local Global Information Fusion](https://mlsanity.com/paper/2207.04535/) (2022)
- [Transformers in Self-Supervised Monocular Depth Estimation with Unknown Camera Intrinsics](https://mlsanity.com/paper/2202.03131/) (2022)
- [GCNDepth: Self-supervised Monocular Depth Estimation based on Graph Convolutional Network](https://mlsanity.com/paper/2112.06782/) (2021)
- [A CNN Based Approach for the Point-Light Photometric Stereo Problem](https://mlsanity.com/paper/2210.04655/) (2022)
- [Depth Estimation with Simplified Transformer](https://mlsanity.com/paper/2204.13791/) (2022)
- [Hybrid Transformer Based Feature Fusion for Self-Supervised Monocular Depth Estimation](https://mlsanity.com/paper/2211.11066/) (2022)

## Abstract (arXiv)

This research presents a novel depth estimation algorithm based on a Transformer-encoder architecture, tailored for the NYU and KITTI Depth Dataset. This research adopts a transformer model, initially renowned for its success in natural language processing, to capture intricate spatial relationships in visual data for depth estimation tasks. A significant innovation of the research is the integration of a composite loss function that combines Structural Similarity Index Measure (SSIM) with Mean Squared Error (MSE). This combined loss function is designed to ensure the structural integrity of the predicted depth maps relative to the original images (via SSIM) while minimizing pixel-wise estimation errors (via MSE). This research approach addresses the challenges of over-smoothing often seen in MSE-based losses and enhances the model's ability to predict depth maps that are not only accurate but also maintain structural coherence with the input images. Through rigorous training and evaluation using the NYU Depth Dataset, the model demonstrates superior performance, marking a significant advancement in single-image depth estimation, particularly in complex indoor and traffic environments.