# Robust Self-Supervised Extrinsic Self-Calibration

arXiv: https://arxiv.org/abs/2308.02153 · HTML: https://mlsanity.com/paper/2308.02153/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 35 references that are in the map.
- First posted on arXiv 4 Aug 2023.

## Builds on (most cited first, 24 of 24)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [Unsupervised Monocular Depth Estimation with Left-Right Consistency](https://mlsanity.com/paper/1609.03677/) (2016)
- [Unsupervised Learning of Depth and Ego-Motion from Video](https://mlsanity.com/paper/1704.07813/) (2017)
- [BARF: Bundle-Adjusting Neural Radiance Fields](https://mlsanity.com/paper/2104.06405/) (2021)
- [3D Packing for Self-Supervised Monocular Depth Estimation](https://mlsanity.com/paper/1905.02693/) (2019)
- [Deeper Depth Prediction with Fully Convolutional Residual Networks](https://mlsanity.com/paper/1606.00373/) (2016)
- [INeRF: Inverting Neural Radiance Fields for Pose Estimation](https://mlsanity.com/paper/2012.05877/) (2020)
- [Self-Calibrating Neural Radiance Fields](https://mlsanity.com/paper/2108.13826/) (2021)
- [Depth from Videos in the Wild: Unsupervised Monocular Depth Learning from Unknown Cameras](https://mlsanity.com/paper/1904.04998/) (2019)
- [RelPose: Predicting Probabilistic Relative Rotation for Single Objects in the Wild](https://mlsanity.com/paper/2208.05963/) (2022)
- [SurroundDepth: Entangling Surrounding Views for Self-Supervised Multi-Camera Depth Estimation](https://mlsanity.com/paper/2204.03636/) (2022)
- [6D Rotation Representation For Unconstrained Head Pose Estimation](https://mlsanity.com/paper/2202.12555/) (2022)
- [360MonoDepth: High-Resolution 360deg Monocular Depth Estimation](https://mlsanity.com/paper/2111.15669/) (2021)
- [Full Surround Monodepth from Multiple Cameras](https://mlsanity.com/paper/2104.00152/) (2021)
- [FisheyeDistanceNet: Self-Supervised Scale-Aware Distance Estimation using Monocular Fisheye Camera for Autonomous Driving](https://mlsanity.com/paper/1910.04076/) (2019)
- [Gradient-based Uncertainty for Monocular Depth Estimation](https://mlsanity.com/paper/2208.02005/) (2022)
- [RM-Depth: Unsupervised Learning of Recurrent Monocular Depth in Dynamic Scenes](https://mlsanity.com/paper/2303.04456/) (2023)
- [Neural Ray Surfaces for Self-Supervised Learning of Depth and Ego-motion](https://mlsanity.com/paper/2008.06630/) (2020)
- [Self-Supervised Camera Self-Calibration from Video](https://mlsanity.com/paper/2112.03325/) (2021)
- [Multi-Camera Collaborative Depth Prediction via Consistent Structure Estimation](https://mlsanity.com/paper/2210.02009/) (2022)
- [Infrastructure-based Multi-Camera Calibration using Radial Projections](https://mlsanity.com/paper/2007.15330/) (2020)
- [TEScalib: Targetless Extrinsic Self-Calibration of LiDAR and Stereo Camera for Automated Driving Vehicles with Uncertainty Analysis](https://mlsanity.com/paper/2202.13847/) (2022)

## Built on by (most cited first, 4 of 4)

- [Delving into Multi-modal Multi-task Foundation Models for Road Scene Understanding: From Learning Paradigm Perspectives](https://mlsanity.com/paper/2402.02968/) (2024)
- [GRIN: Zero-Shot Metric Depth with Pixel-Level Diffusion](https://mlsanity.com/paper/2409.09896/) (2024)
- [Self-Supervised Geometry-Guided Initialization for Robust Monocular Visual Odometry](https://mlsanity.com/paper/2406.00929/) (2024)
- [Multi-Camera Self-Calibration in Sports Motion Capture: Leveraging Human and Stick Poses](https://mlsanity.com/paper/2604.17567/) (2026)

## Abstract (arXiv)

Autonomous vehicles and robots need to operate over a wide variety of scenarios in order to complete tasks efficiently and safely. Multi-camera self-supervised monocular depth estimation from videos is a promising way to reason about the environment, as it generates metrically scaled geometric predictions from visual data without requiring additional sensors. However, most works assume well-calibrated extrinsics to fully leverage this multi-camera setup, even though accurate and efficient calibration is still a challenging problem. In this work, we introduce a novel method for extrinsic calibration that builds upon the principles of self-supervised monocular depth and ego-motion learning. Our proposed curriculum learning strategy uses monocular depth and pose estimators with velocity supervision to estimate extrinsics, and then jointly learns extrinsic calibration along with depth and pose for a set of overlapping cameras rigidly attached to a moving vehicle. Experiments on a benchmark multi-camera dataset (DDAD) demonstrate that our method enables self-calibration in various scenes robustly and efficiently compared to a traditional vision-based pose estimation pipeline. Furthermore, we demonstrate the benefits of extrinsics self-calibration as a way to improve depth prediction via joint optimization.