# Depth Estimation with Simplified Transformer

arXiv: https://arxiv.org/abs/2204.13791 · HTML: https://mlsanity.com/paper/2204.13791/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 33 references that are in the map.
- First posted on arXiv 28 Apr 2022.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [Unsupervised Monocular Depth Estimation with Left-Right Consistency](https://mlsanity.com/paper/1609.03677/) (2016)
- [Unsupervised Learning of Depth and Ego-Motion from Video](https://mlsanity.com/paper/1704.07813/) (2017)
- [3D Packing for Self-Supervised Monocular Depth Estimation](https://mlsanity.com/paper/1905.02693/) (2019)
- [Unsupervised Scale-consistent Depth and Ego-motion Learning from Monocular Video](https://mlsanity.com/paper/1908.10553/) (2019)
- [Revisiting Stereo Depth Estimation From a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2011.02910/) (2020)
- [Unsupervised Learning of Depth and Ego-Motion from Monocular Video Using 3D Geometric Constraints](https://mlsanity.com/paper/1802.05522/) (2018)
- [Learning Depth from Single Monocular Images Using Deep Convolutional Neural Fields](https://mlsanity.com/paper/1502.07411/) (2015)
- [Learning Depth from Monocular Videos using Direct Methods](https://mlsanity.com/paper/1712.00175/) (2017)
- [Efficient Self-supervised Vision Transformers for Representation Learning](https://mlsanity.com/paper/2106.09785/) (2021)
- [Towards Better Generalization: Joint Depth-Pose Learning without PoseNet](https://mlsanity.com/paper/2004.01314/) (2020)
- [Learning Efficient Vision Transformers via Fine-Grained Manifold Distillation](https://mlsanity.com/paper/2107.01378/) (2021)

## Built on by (most cited first, 8 of 8)

- [Image Masking for Robust Self-Supervised Monocular Depth Estimation](https://mlsanity.com/paper/2210.02357/) (2022)
- [DeLiRa: Self-Supervised Depth, Light, and Radiance Fields](https://mlsanity.com/paper/2304.02797/) (2023)
- [Towards Scale-Aware Full Surround Monodepth with Transformers](https://mlsanity.com/paper/2407.10406/) (2024)
- [A Novel Spike Transformer Network for Depth Estimation from Event Cameras via Cross-modality Knowledge Distillation](https://mlsanity.com/paper/2404.17335/) (2024)
- [FCN+: Global Receptive Convolution Makes FCN Great Again](https://mlsanity.com/paper/2303.04589/) (2023)
- [Swin-Free: Achieving Better Cross-Window Attention and Efficiency with Size-varying Window](https://mlsanity.com/paper/2306.13776/) (2023)
- [Depth Estimation Algorithm Based on Transformer-Encoder and Feature Fusion](https://mlsanity.com/paper/2403.01370/) (2024)
- [ReduceFormer: Attention with Tensor Reduction by Summation](https://mlsanity.com/paper/2406.07488/) (2024)

## Abstract (arXiv)

Transformer and its variants have shown state-of-the-art results in many vision tasks recently, ranging from image classification to dense prediction. Despite of their success, limited work has been reported on improving the model efficiency for deployment in latency-critical applications, such as autonomous driving and robotic navigation. In this paper, we aim at improving upon the existing transformers in vision, and propose a method for self-supervised monocular Depth Estimation with Simplified Transformer (DEST), which is efficient and particularly suitable for deployment on GPU-based platforms. Through strategic design choices, our model leads to significant reduction in model size, complexity, as well as inference latency, while achieving superior accuracy as compared to state-of-the-art. We also show that our design generalize well to other dense prediction task without bells and whistles.