# LS-Net: Learning to Solve Nonlinear Least Squares for Monocular Stereo

arXiv: https://arxiv.org/abs/1809.02966 · HTML: https://mlsanity.com/paper/1809.02966/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Sep 2018.

## Built on by (most cited first, 28 of 28)

- [DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras](https://mlsanity.com/paper/2108.10869/) (2021)
- [Pixel-Perfect Structure-from-Motion with Featuremetric Refinement](https://mlsanity.com/paper/2108.08291/) (2021)
- [Back to the Feature: Learning Robust Camera Localization from Pixels to Pose](https://mlsanity.com/paper/2103.09213/) (2021)
- [Theseus: A Library for Differentiable Nonlinear Optimization](https://mlsanity.com/paper/2207.09442/) (2022)
- [RePOSE: Fast 6D Object Pose Refinement via Deep Texture Rendering](https://mlsanity.com/paper/2104.00633/) (2021)
- [FlowMap: High-Quality Camera Poses, Intrinsics, and Depth via Gradient Descent](https://mlsanity.com/paper/2404.15259/) (2024)
- [GeoCalib: Learning Single-image Calibration with Geometric Optimization](https://mlsanity.com/paper/2409.06704/) (2024)
- [Deep Two-View Structure-from-Motion Revisited](https://mlsanity.com/paper/2104.00556/) (2021)
- [Satellite Image Based Cross-view Localization for Autonomous Vehicle](https://mlsanity.com/paper/2207.13506/) (2022)
- [VOLDOR: Visual Odometry from Log-logistic Dense Optical flow Residuals](https://mlsanity.com/paper/2104.06789/) (2021)
- [Learning to Fit Morphable Models](https://mlsanity.com/paper/2111.14824/) (2021)
- [Deep Learning for Visual Localization and Mapping: A Survey](https://mlsanity.com/paper/2308.14039/) (2023)
- [MetaPose: Fast 3D Pose from Multiple Views without 3D Supervision](https://mlsanity.com/paper/2108.04869/) (2021)
- [DRO: Deep Recurrent Optimizer for Video to Depth](https://mlsanity.com/paper/2103.13201/) (2021)
- [RAGO: Recurrent Graph Optimizer For Multiple Rotation Averaging](https://mlsanity.com/paper/2212.07211/) (2022)
- [Fusing the Old with the New: Learning Relative Camera Pose with Geometry-Guided Uncertainty](https://mlsanity.com/paper/2104.08278/) (2021)
- [Level-S2fM: Structure from Motion on Neural Level Set of Implicit Surfaces](https://mlsanity.com/paper/2211.12018/) (2022)
- [RL-PGO: Reinforcement Learning-based Planar Pose-Graph Optimization](https://mlsanity.com/paper/2202.13221/) (2022)
- [A Game of Bundle Adjustment -- Learning Efficient Convergence](https://mlsanity.com/paper/2308.13270/) (2023)
- [DeepMLE: A Robust Deep Maximum Likelihood Estimator for Two-view Structure from Motion](https://mlsanity.com/paper/2210.05517/) (2022)
- [Uncertainty-Driven Dense Two-View Structure from Motion](https://mlsanity.com/paper/2302.00523/) (2023)
- [On the Coupling of Depth and Egomotion Networks for Self-Supervised Structure from Motion](https://mlsanity.com/paper/2106.04007/) (2021)
- [Multi-view Monocular Depth and Uncertainty Prediction with Deep SfM in Dynamic Environments](https://mlsanity.com/paper/2201.08633/) (2022)
- [TransPoser: Transformer as an Optimizer for Joint Object Shape and Pose Estimation](https://mlsanity.com/paper/2303.13477/) (2023)
- [ADU-Depth: Attention-based Distillation with Uncertainty Modeling for Depth Estimation](https://mlsanity.com/paper/2309.14744/) (2023)
- [Revisit Self-supervised Depth Estimation with Local Structure-from-Motion](https://mlsanity.com/paper/2407.19166/) (2024)
- [Str-L Pose: Integrating Point and Structured Line for Relative Pose Estimation in Dual-Graph](https://mlsanity.com/paper/2408.15750/) (2024)
- [SIRE: SE(3) Intrinsic Rigidity Embeddings](https://mlsanity.com/paper/2503.07739/) (2025)

## Abstract (arXiv)

Sum-of-squares objective functions are very popular in computer vision algorithms. However, these objective functions are not always easy to optimize. The underlying assumptions made by solvers are often not satisfied and many problems are inherently ill-posed. In this paper, we propose LS-Net, a neural nonlinear least squares optimization algorithm which learns to effectively optimize these cost functions even in the presence of adversities. Unlike traditional approaches, the proposed solver requires no hand-crafted regularizers or priors as these are implicitly learned from the data. We apply our method to the problem of motion stereo ie. jointly estimating the motion and scene geometry from pairs of images of a monocular sequence. We show that our learned optimizer is able to efficiently and effectively solve this challenging optimization problem.