# Neural Ray Surfaces for Self-Supervised Learning of Depth and Ego-motion

arXiv: https://arxiv.org/abs/2008.06630 · HTML: https://mlsanity.com/paper/2008.06630/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Aug 2020.

## Built on by (most cited first, 20 of 20)

- [MapAnything: Universal Feed-Forward Metric 3D Reconstruction](https://mlsanity.com/paper/2509.13414/) (2025)
- [Cameras as Rays: Pose Estimation via Ray Diffusion](https://mlsanity.com/paper/2402.14817/) (2024)
- [Multi-Frame Self-Supervised Depth with Transformers](https://mlsanity.com/paper/2204.07616/) (2022)
- [Full Surround Monodepth from Multiple Cameras](https://mlsanity.com/paper/2104.00152/) (2021)
- [Geometric Unsupervised Domain Adaptation for Semantic Segmentation](https://mlsanity.com/paper/2103.16694/) (2021)
- [Learning Optical Flow, Depth, and Scene Flow without Real-World Labels](https://mlsanity.com/paper/2203.15089/) (2022)
- [Attentive and Contrastive Learning for Joint Depth and Motion Field Estimation](https://mlsanity.com/paper/2110.06853/) (2021)
- [Self-Supervised Camera Self-Calibration from Video](https://mlsanity.com/paper/2112.03325/) (2021)
- [Depth Field Networks for Generalizable Multi-view Scene Representation](https://mlsanity.com/paper/2207.14287/) (2022)
- [DualRefine: Self-Supervised Depth and Pose Estimation Through Iterative Epipolar Sampling and Refinement Toward Equilibrium](https://mlsanity.com/paper/2304.03560/) (2023)
- [DeLiRa: Self-Supervised Depth, Light, and Radiance Fields](https://mlsanity.com/paper/2304.02797/) (2023)
- [Depth360: Self-supervised Learning for Monocular Depth Estimation using Learnable Camera Distortion Model](https://mlsanity.com/paper/2110.10415/) (2021)
- [Robust Self-Supervised Extrinsic Self-Calibration](https://mlsanity.com/paper/2308.02153/) (2023)
- [A Comprehensive Overview of Fish-Eye Camera Distortion Correction Methods](https://mlsanity.com/paper/2401.00442/) (2023)
- [CalibAnyView: Beyond Single-View Camera Calibration in the Wild](https://mlsanity.com/paper/2605.14615/) (2026)
- [CylinderDepth: Cylindrical Spatial Attention for Multi-View Consistent Self-Supervised Surround Depth Estimation](https://mlsanity.com/paper/2511.16428/) (2025)
- [Keypoints Tracking via Transformer Networks](https://mlsanity.com/paper/2203.12848/) (2022)
- [Neural Camera Models](https://mlsanity.com/paper/2208.12903/) (2022)
- [Staircase Localization for Autonomous Exploration in Urban Environments](https://mlsanity.com/paper/2403.17330/) (2024)
- [CrossDepth: Geometry-Constrained Attention for Generalizable Multi-View Surround Depth Estimation](https://mlsanity.com/paper/2609.05397/) (2026)

## Abstract (arXiv)

Self-supervised learning has emerged as a powerful tool for depth and ego-motion estimation, leading to state-of-the-art results on benchmark datasets. However, one significant limitation shared by current methods is the assumption of a known parametric camera model -- usually the standard pinhole geometry -- leading to failure when applied to imaging systems that deviate significantly from this assumption (e.g., catadioptric cameras or underwater imaging). In this work, we show that self-supervision can be used to learn accurate depth and ego-motion estimation without prior knowledge of the camera model. Inspired by the geometric model of Grossberg and Nayar, we introduce Neural Ray Surfaces (NRS), convolutional networks that represent pixel-wise projection rays, approximating a wide range of cameras. NRS are fully differentiable and can be learned end-to-end from unlabeled raw videos. We demonstrate the use of NRS for self-supervised learning of visual odometry and depth estimation from raw videos obtained using a wide variety of camera systems, including pinhole, fisheye, and catadioptric.