# Multi-View Stereo by Temporal Nonparametric Fusion

arXiv: https://arxiv.org/abs/1904.06397 · HTML: https://mlsanity.com/paper/1904.06397/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Apr 2019.

## Built on by (most cited first, 30 of 46)

- [NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Video](https://mlsanity.com/paper/2104.00681/) (2021)
- [NerfingMVS: Guided Optimization of Neural Radiance Fields for Indoor Multi-view Stereo](https://mlsanity.com/paper/2109.01129/) (2021)
- [The Temporal Opportunist: Self-Supervised Multi-Frame Monocular Depth](https://mlsanity.com/paper/2104.14540/) (2021)
- [Time Will Tell: New Outlooks and A Baseline for Temporal Multi-View 3D Object Detection](https://mlsanity.com/paper/2210.02443/) (2022)
- [TransformerFusion: Monocular RGB Scene Reconstruction using Transformers](https://mlsanity.com/paper/2107.02191/) (2021)
- [SimpleRecon: 3D Reconstruction Without 3D Convolutions](https://mlsanity.com/paper/2208.14743/) (2022)
- [VolumeFusion: Deep Depth Fusion for 3D Scene Reconstruction](https://mlsanity.com/paper/2108.08623/) (2021)
- [TANDEM: Tracking and Dense Mapping in Real-time using Deep Multi-view Stereo](https://mlsanity.com/paper/2111.07418/) (2021)
- [VoRTX: Volumetric 3D Reconstruction With Transformers for Voxelwise View Selection and Fusion](https://mlsanity.com/paper/2112.00236/) (2021)
- [PlanarRecon: Real-time 3D Plane Detection and Reconstruction from Posed Monocular Videos](https://mlsanity.com/paper/2206.07710/) (2022)
- [Neural Radiance Fields Approach to Deep Multi-View Photometric Stereo](https://mlsanity.com/paper/2110.05594/) (2021)
- [3DVNet: Multi-View Depth Prediction and Volumetric Refinement](https://mlsanity.com/paper/2112.00202/) (2021)
- [MACARONS: Mapping And Coverage Anticipation with RGB Online Self-Supervision](https://mlsanity.com/paper/2303.03315/) (2023)
- [Learning-based Multi-View Stereo: A Survey](https://mlsanity.com/paper/2408.15235/) (2024)
- [MonoNeuralFusion: Online Monocular Neural 3D Reconstruction with Geometric Priors](https://mlsanity.com/paper/2209.15153/) (2022)
- [FineRecon: Depth-aware Feed-forward Network for Detailed 3D Reconstruction](https://mlsanity.com/paper/2304.01480/) (2023)
- [CVRecon: Rethinking 3D Geometric Feature Learning For Neural Reconstruction](https://mlsanity.com/paper/2304.14633/) (2023)
- [ChiTransformer:Towards Reliable Stereo from Cues](https://mlsanity.com/paper/2203.04554/) (2022)
- [A Confidence-based Iterative Solver of Depths and Surface Normals for Deep Multi-view Stereo](https://mlsanity.com/paper/2201.07609/) (2022)
- [MonoNeRF: Learning Generalizable NeRFs from Monocular Videos without Camera Pose](https://mlsanity.com/paper/2210.07181/) (2022)
- [DoubleTake: Geometry Guided Depth Estimation](https://mlsanity.com/paper/2406.18387/) (2024)
- [3D Former: Monocular Scene Reconstruction with 3D SDF Transformers](https://mlsanity.com/paper/2301.13510/) (2023)
- [VisFusion: Visibility-aware Online 3D Scene Reconstruction from Videos](https://mlsanity.com/paper/2304.10687/) (2023)
- [DurLAR: A High-fidelity 128-channel LiDAR Dataset with Panoramic Ambient and Reflectivity Imagery for Multi-modal Autonomous Driving Applications](https://mlsanity.com/paper/2406.10068/) (2024)
- [Incremental Dense Reconstruction from Monocular Video with Guided Sparse Feature Volume Fusion](https://mlsanity.com/paper/2305.14918/) (2023)
- [MonoSelfRecon: Purely Self-Supervised Explicit Generalizable 3D Reconstruction of Indoor Scenes from Monocular RGB Views](https://mlsanity.com/paper/2404.06753/) (2024)
- [Online Time Series Anomaly Detection with State Space Gaussian Processes](https://mlsanity.com/paper/2201.06763/) (2022)
- [NeuralRoom: Geometry-Constrained Neural Implicit Surfaces for Indoor Scene Reconstruction](https://mlsanity.com/paper/2210.06853/) (2022)
- [Virtual Occlusions Through Implicit Depth](https://mlsanity.com/paper/2305.07014/) (2023)
- [Unified Scene Representation and Reconstruction for 3D Large Language Models](https://mlsanity.com/paper/2404.13044/) (2024)

## Abstract (arXiv)

We propose a novel idea for depth estimation from multi-view image-pose pairs, where the model has capability to leverage information from previous latent-space encodings of the scene. This model uses pairs of images and poses, which are passed through an encoder--decoder model for disparity estimation. The novelty lies in soft-constraining the bottleneck layer by a nonparametric Gaussian process prior. We propose a pose-kernel structure that encourages similar poses to have resembling latent spaces. The flexibility of the Gaussian process (GP) prior provides adapting memory for fusing information from previous views. We train the encoder--decoder and the GP hyperparameters jointly end-to-end. In addition to a batch method, we derive a lightweight estimation scheme that circumvents standard pitfalls in scaling Gaussian process inference, and demonstrate how our scheme can run in real-time on smart devices.