mlsanityData to 14 Oct 2026

SwinMTL: A Shared Architecture for Simultaneous Depth Estimation and Semantic Segmentation from Monocular Camera Images

arXiv 2403.10662 ↗First posted 15 Mar 2024cs.CVcited by 13

Not on the map. Why?

Abstract (arXiv)

This research paper presents an innovative multi-task learning framework that allows concurrent depth estimation and semantic segmentation using a single camera. The proposed approach is based on a shared encoder-decoder architecture, which integrates various techniques to improve the accuracy of the depth estimation and semantic segmentation task without compromising computational efficiency. Additionally, the paper incorporates an adversarial training component, employing a Wasserstein GAN framework with a critic network, to refine model's predictions. The framework is thoroughly evaluated on two datasets - the outdoor Cityscapes dataset and the indoor NYU Depth V2 dataset - and it outperforms existing state-of-the-art methods in both segmentation and depth estimation tasks. We also conducted ablation studies to analyze the contributions of different components, including pre-training strategies, the inclusion of critics, the use of logarithmic depth scaling, and advanced image augmentations, to provide a better understanding of the proposed framework. The accompanying source code is accessible at urlhttps://github.com/PardisTaghavi/SwinMTL.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Builds on 38 of its 55 references that are in the map. First posted on arXiv 15 Mar 2024.

Builds on

38 of 55 references.

References · 38

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  4. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,859
  5. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  6. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,241
  7. 2017Improved Training of Wasserstein GANscited by 2,235
  8. 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,358
  9. 2014Depth Map Prediction from a Single Image using a Multi-Scale Deep Networkcited by 1,274
  10. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 834
  11. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 821
  12. 2018Deep Ordinal Regression Network for Monocular Depth Estimationcited by 723
26 more
  1. 2017Wasserstein GANcited by 639
  2. 2020AdaBins: Depth Estimation using Adaptive Binscited by 601
  3. 2018End-to-End Multi-Task Learning with Attentioncited by 534
  4. 2016Cross-stitch Networks for Multi-task Learningcited by 467
  5. 2018PAD-Net: Multi-Tasks Guided Prediction-and-Distillation Network for Simultaneous Depth Estimation and Scene Parsingcited by 224
  6. 2022Practical Stereo Matching via Cascaded Recurrent Network with Adaptive Correlationcited by 197
  7. 2020MTI-Net: Multi-Scale Task Interaction Networks for Multi-Task Learningcited by 157
  8. 2022Revealing the Dark Secrets of Masked Image Modelingcited by 117
  9. 2016MultiNet: Real-time Joint Semantic Reasoning for Autonomous Drivingcited by 90
  10. 2022InvPT: Inverted Pyramid Multi-task Transformer for Dense Scene Understandingcited by 89
  11. 2020Unsupervised Monocular Depth Learning in Dynamic Scenescited by 84
  12. 2023Semantic Segmentation using Vision Transformers: A surveycited by 73
  13. 2021Exploring Relational Context for Multi-Task Dense Predictioncited by 71
  14. 2020Three Ways to Improve Semantic Segmentation with Self-Supervised Depth Estimationcited by 66
  15. 2021Learning Monocular Depth in Dynamic Scenes via Instance-Aware Projection Consistencycited by 58
  16. 2019Unsupervised Monocular Depth and Ego-motion Learning with Structure and Semanticscited by 49
  17. 2022Disentangling Object Motion and Occlusion for Unsupervised Multi-frame Monocular Depthcited by 48
  18. 2018Unsupervised Adversarial Depth Estimation using Cycled Generative Networkscited by 45
  19. 2016Joint Semantic Segmentation and Depth Estimation with Deep Convolutional Networkscited by 36
  20. 2023Zero-Shot Metric Depth with a Field-of-View Conditioned Diffusion Modelcited by 29
  21. 2022MGNet: Monocular Geometric Scene Understanding for Autonomous Drivingcited by 27
  22. 2022DenseMTL: Cross-task Attention Mechanism for Dense Multi-task Learningcited by 25
  23. 2021Attentive and Contrastive Learning for Joint Depth and Motion Field Estimationcited by 24
  24. 2022OpenMixup: Open Mixup Toolbox and Benchmark for Visual Representation Learningcited by 21
  25. 2020Multi-task GANs for Semantic Segmentation and Depth Completion with Cycle Consistencycited by 18
  26. 2024Hybridnet for depth estimation and semantic segmentationcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 4; 2026: 7.

13 papers.

Sort:

By area:

  1. 2024StitchFusion: Weaving Any Visual Modalities to Enhance Multimodal Semantic Segmentationcited by 39
  2. 2024TwinLiteNet+: An Enhanced Multi-Task Segmentation Model for Autonomous Drivingcited by 5
  3. 2026Spatially Aware World Action Model via Geometric Latent Diffusioncited by 5
  4. 2026NaviDriveVLM: Decoupling High-Level Reasoning and Motion Planning for Autonomous Drivingcited by 4
  5. 2025EvidMTL: Evidential Multi-Task Learning for Uncertainty-Aware Semantic Surface Mapping from Monocular RGB Imagescited by 3
  6. 2025M2H: Multi-Task Learning with Efficient Window-Based Cross-Task Attention for Monocular Spatial Perceptioncited by 3
  7. 2025Enhancing Monocular Depth Estimation with Multi-Source Auxiliary Taskscited by 2
  8. 2025Multi-task Learning For Joint Action and Gesture Recognitioncited by 1
  9. 2026DenseMLLM: Standard Multimodal LLMs for Dense Predictioncited by 0
  10. 2026B3-Net: Controlled Posterior Bridge Learning for Multi-Task Dense Predictioncited by 0
  11. 2026DepthPolyp: Pseudo-Depth Guided Lightweight Segmentation for Real-Time Colonoscopycited by 0
  12. 2026Mono-Hydra++: Real-Time Monocular Scene Graph Construction with Multi-Task Learning for 3D Indoor Mappingcited by 0
1 more
  1. 2026DPNeXt: A Lightweight Multi-Scale Feature Fusion Framework for Efficient ViT-Based Multi-Task Dense Predictioncited by 0