mlsanityData to 7 Oct 2026

Transformers in Unsupervised Structure-from-Motion

arXiv 2312.10529 ↗First posted 16 Dec 2023cs.CV cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Transformers have revolutionized deep learning based computer vision with improved performance as well as robustness to natural corruptions and adversarial attacks. Transformers are used predominantly for 2D vision tasks, including image classification, semantic segmentation, and object detection. However, robots and advanced driver assistance systems also require 3D scene understanding for decision making by extracting structure-from-motion (SfM). We propose a robust transformer-based monocular SfM method that learns to predict monocular pixel-wise depth, ego vehicle's translation and rotation, as well as camera's focal length and principal point, simultaneously. With experiments on KITTI and DDAD datasets, we demonstrate how to adapt different vision transformers and compare them against contemporary CNN-based methods. Our study shows that transformer-based architecture, though lower in run-time efficiency, achieves comparable performance while being more robust against natural corruptions, as well as untargeted and targeted attacks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 44 of its 60 references that are in the map. First posted on arXiv 16 Dec 2023.

Builds on

44 of 60 references.

References · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2020End-to-End Object Detection with Transformerscited by 7,384
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  9. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  10. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  11. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  12. 2020Shortcut Learning in Deep Neural Networkscited by 1,928
32 more
  1. 2021Vision Transformers for Dense Predictioncited by 1,789
  2. 2016Adversarial examples in the physical worldcited by 1,762
  3. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
  4. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  5. 2014Depth Map Prediction from a Single Image using a Multi-Scale Deep Networkcited by 1,264
  6. 2021Segmenter: Transformer for Semantic Segmentationcited by 855
  7. 2016Unsupervised Monocular Depth Estimation with Left-Right Consistencycited by 819
  8. 2017Unsupervised Learning of Depth and Ego-Motion from Videocited by 778
  9. 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 749
  10. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 671
  11. 2016RefineNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentationcited by 463
  12. 20193D Packing for Self-Supervised Monocular Depth Estimationcited by 437
  13. 2019Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Comingcited by 332
  14. 2021Understanding Robustness of Transformers for Image Classificationcited by 250
  15. 2019Unsupervised Scale-consistent Depth and Ego-motion Learning from Monocular Videocited by 244
  16. 2018Competitive Collaboration: Joint Unsupervised Learning of Depth, Camera Motion, Optical Flow and Motion Segmentationcited by 222
  17. 2021Vision Transformers are Robust Learnerscited by 208
  18. 2020On the uncertainty of self-supervised monocular depth estimationcited by 182
  19. 2020Self-Supervised Monocular Depth Estimation: Solving the Dynamic Object Problem by Semantic Guidancecited by 148
  20. 2018Evaluation of CNN-based Single-Image Depth Estimation Methodscited by 137
  21. 2021Transformer-Based Attention Networks for Continuous Pixel-Wise Predictioncited by 125
  22. 2020Self-supervised Monocular Trained Depth Estimation using Self-attention and Discrete Disparity Volumecited by 116
  23. 2020HR-Depth: High Resolution Self-Supervised Monocular Depth Estimationcited by 115
  24. 2021Unsupervised Scale-consistent Depth Learning from Videocited by 69
  25. 2020Targeted Adversarial Perturbations for Monocular Depth Predictioncited by 40
  26. 2022Deep Digging into the Generalization of Self-Supervised Monocular Depth Estimationcited by 24
  27. 2022Transformers in Self-Supervised Monocular Depth Estimation with Unknown Camera Intrinsicscited by 19
  28. 2021Multimodal Scale Consistency and Awareness for Monocular Self-Supervised Depth Estimationcited by 16
  29. 2019A 3D-Deep-Learning-based Augmented Reality Calibration Method for Robotic Environments using Depth Sensor Datacited by 10
  30. 2022A Comprehensive Study of Vision Transformers on Dense Prediction Taskscited by 8
  31. 2022On the interplay of adversarial robustness and architecture components: patches, convolution and attentioncited by 4
  32. 2020Crowdsourced 3D Mapping: A Combined Multi-View Geometry and Self-Supervised Learning Approachcited by 3

Built on by

Not cited yet.