# LEGO: Learning Edge with Geometry all at Once by Watching Videos

arXiv: https://arxiv.org/abs/1803.05648 · HTML: https://mlsanity.com/paper/1803.05648/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Mar 2018.

## Built on by (most cited first, 30 of 47)

- [The Temporal Opportunist: Self-Supervised Multi-Frame Monocular Depth](https://mlsanity.com/paper/2104.14540/) (2021)
- [MonoViT: Self-Supervised Monocular Depth Estimation with a Vision Transformer](https://mlsanity.com/paper/2208.03543/) (2022)
- [P3Depth: Monocular Depth Estimation with a Piecewise Planarity Prior](https://mlsanity.com/paper/2204.02091/) (2022)
- [Self-Supervised Monocular Depth and Ego-Motion Estimation in Endoscopy: Appearance Flow to the Rescue](https://mlsanity.com/paper/2112.08122/) (2021)
- [Channel-Wise Attention-Based Network for Self-Supervised Monocular Depth Estimation](https://mlsanity.com/paper/2112.13047/) (2021)
- [Learning Monocular Depth in Dynamic Scenes via Instance-Aware Projection Consistency](https://mlsanity.com/paper/2102.02629/) (2021)
- [Disentangling Object Motion and Occlusion for Unsupervised Multi-frame Monocular Depth](https://mlsanity.com/paper/2203.15174/) (2022)
- [Multi-View Depth Estimation by Fusing Single-View Depth Probability with Multi-View Geometry](https://mlsanity.com/paper/2112.08177/) (2021)
- [StructDepth: Leveraging the structural regularities for self-supervised indoor depth estimation](https://mlsanity.com/paper/2108.08574/) (2021)
- [On the Importance of Accurate Geometry Data for Dense 3D Vision Tasks](https://mlsanity.com/paper/2303.14840/) (2023)
- [Attentive and Contrastive Learning for Joint Depth and Motion Field Estimation](https://mlsanity.com/paper/2110.06853/) (2021)
- [Is my Depth Ground-Truth Good Enough? HAMMER -- Highly Accurate Multi-Modal Dataset for DEnse 3D Scene Regression](https://mlsanity.com/paper/2205.04565/) (2022)
- [AugUndo: Scaling Up Augmentations for Monocular Depth Completion and Estimation](https://mlsanity.com/paper/2310.09739/) (2023)
- [Advancing Self-supervised Monocular Depth Learning with Sparse LiDAR](https://mlsanity.com/paper/2109.09628/) (2021)
- [GasMono: Geometry-Aided Self-Supervised Monocular Depth Estimation for Indoor Scenes](https://mlsanity.com/paper/2309.16019/) (2023)
- [GCNDepth: Self-supervised Monocular Depth Estimation based on Graph Convolutional Network](https://mlsanity.com/paper/2112.06782/) (2021)
- [3D Hierarchical Refinement and Augmentation for Unsupervised Learning of Depth and Pose from Monocular Video](https://mlsanity.com/paper/2112.03045/) (2021)
- [MonoNeRF: Learning Generalizable NeRFs from Monocular Videos without Camera Pose](https://mlsanity.com/paper/2210.07181/) (2022)
- [SwinDepth: Unsupervised Depth Estimation using Monocular Sequences via Swin Transformer and Densely Cascaded Network](https://mlsanity.com/paper/2301.06715/) (2023)
- [Self-Supervised Monocular Depth Estimation with Self-Reference Distillation and Disparity Offset Refinement](https://mlsanity.com/paper/2302.09789/) (2023)
- [WS-SfMLearner: Self-supervised Monocular Depth and Ego-motion Estimation on Surgical Videos with Unknown Camera Parameters](https://mlsanity.com/paper/2308.11776/) (2023)
- [NDDepth: Normal-Distance Assisted Monocular Depth Estimation and Completion](https://mlsanity.com/paper/2311.07166/) (2023)
- [Extending Foundational Monocular Depth Estimators to Fisheye Cameras with Calibration Tokens](https://mlsanity.com/paper/2508.04928/) (2025)
- [Visual Attention-based Self-supervised Absolute Depth Estimation using Geometric Priors in Autonomous Driving](https://mlsanity.com/paper/2205.08780/) (2022)
- [Unsupervised OmniMVS: Efficient Omnidirectional Depth Inference via Establishing Pseudo-Stereo Supervision](https://mlsanity.com/paper/2302.09922/) (2023)
- [A Surface-normal Based Neural Framework for Colonoscopy Reconstruction](https://mlsanity.com/paper/2303.07264/) (2023)
- [DS-Depth: Dynamic and Static Depth Estimation via a Fusion Cost Volume](https://mlsanity.com/paper/2308.07225/) (2023)
- [ColDE: A Depth Estimation Framework for Colonoscopy Reconstruction](https://mlsanity.com/paper/2111.10371/) (2021)
- [Unsupervised Learning of Monocular Depth and Ego-Motion Using Multiple Masks](https://mlsanity.com/paper/2104.00431/) (2021)
- [MonoPP: Metric-Scaled Self-Supervised Monocular Depth Estimation by Planar-Parallax Geometry in Automotive Applications](https://mlsanity.com/paper/2411.19717/) (2024)

## Abstract (arXiv)

Learning to estimate 3D geometry in a single image by watching unlabeled videos via deep convolutional network is attracting significant attention. In this paper, we introduce a "3D as-smooth-as-possible (3D-ASAP)" prior inside the pipeline, which enables joint estimation of edges and 3D scene, yielding results with significant improvement in accuracy for fine detailed structures. Specifically, we define the 3D-ASAP prior by requiring that any two points recovered in 3D from an image should lie on an existing planar surface if no other cues provided. We design an unsupervised framework that Learns Edges and Geometry (depth, normal) all at Once (LEGO). The predicted edges are embedded into depth and surface normal smoothness terms, where pixels without edges in-between are constrained to satisfy the prior. In our framework, the predicted depths, normals and edges are forced to be consistent all the time. We conduct experiments on KITTI to evaluate our estimated geometry and CityScapes to perform edge evaluation. We show that in all of the tasks, i.e.depth, normal and edge, our algorithm vastly outperforms other state-of-the-art (SOTA) algorithms, demonstrating the benefits of our approach.