# The Implicit Values of A Good Hand Shake: Handheld Multi-Frame Neural Depth Refinement

arXiv: https://arxiv.org/abs/2111.13738 · HTML: https://mlsanity.com/paper/2111.13738/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 62 references that are in the map.
- First posted on arXiv 26 Nov 2021.

## Builds on (most cited first, 25 of 25)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [Nerfies: Deformable Neural Radiance Fields](https://mlsanity.com/paper/2011.12948/) (2020)
- [ORB-SLAM2: an Open-Source SLAM System for Monocular, Stereo and RGB-D Cameras](https://mlsanity.com/paper/1610.06475/) (2016)
- [MVSNet: Depth Inference for Unstructured Multi-view Stereo](https://mlsanity.com/paper/1804.02505/) (2018)
- [MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo](https://mlsanity.com/paper/2103.15595/) (2021)
- [Depth-supervised NeRF: Fewer Views and Faster Training for Free](https://mlsanity.com/paper/2107.02791/) (2021)
- [Pyramid Stereo Matching Network](https://mlsanity.com/paper/1803.08669/) (2018)
- [End-to-End Learning of Geometry and Context for Deep Stereo Regression](https://mlsanity.com/paper/1703.04309/) (2017)
- [RAFT-Stereo: Multilevel Recurrent Field Transforms for Stereo Matching](https://mlsanity.com/paper/2109.07547/) (2021)
- [Depth-Aware Video Frame Interpolation](https://mlsanity.com/paper/1904.00830/) (2019)
- [Consistent Video Depth Estimation](https://mlsanity.com/paper/2004.15021/) (2020)
- [Single Image Portrait Relighting](https://mlsanity.com/paper/1905.00824/) (2019)
- [Rethinking RGB-D Salient Object Detection: Models, Data Sets, and Large-Scale Benchmarks](https://mlsanity.com/paper/1907.06781/) (2019)
- [HITNet: Hierarchical Iterative Tile Refinement Network for Real-time Stereo Matching](https://mlsanity.com/paper/2007.12140/) (2020)
- [Handheld Multi-Frame Super-Resolution](https://mlsanity.com/paper/1905.03277/) (2019)
- [MobileStereoNet: Towards Lightweight Deep Networks for Stereo Matching](https://mlsanity.com/paper/2108.09770/) (2021)
- [Mobile Computational Photography: A Tour](https://mlsanity.com/paper/2102.09000/) (2021)
- [Deformable Kernel Networks for Joint Image Filtering](https://mlsanity.com/paper/1910.08373/) (2019)
- [3D Object Proposals using Stereo Imagery for Accurate Object Class Detection](https://mlsanity.com/paper/1608.07711/) (2016)
- [Consistent Depth of Moving Objects in Video](https://mlsanity.com/paper/2108.01166/) (2021)
- [A megapixel time-gated SPAD image sensor for 2D and 3D imaging applications](https://mlsanity.com/paper/1912.12910/) (2019)
- [RGB-D Object Detection and Semantic Segmentation for Autonomous Manipulation in Clutter](https://mlsanity.com/paper/1810.00818/) (2018)
- [BTS-Net: Bi-directional Transfer-and-Selection Network For RGB-D Salient Object Detection](https://mlsanity.com/paper/2104.01784/) (2021)

## Built on by (most cited first, 13 of 13)

- [GenSDF: Two-Stage Learning of Generalizable Signed Distance Functions](https://mlsanity.com/paper/2206.02780/) (2022)
- [Transient Neural Radiance Fields for Lidar View Synthesis and 3D Reconstruction](https://mlsanity.com/paper/2307.09555/) (2023)
- [Consistent Direct Time-of-Flight Video Depth Super-Resolution](https://mlsanity.com/paper/2211.08658/) (2022)
- [Neural Spline Fields for Burst Image Fusion and Layer Separation](https://mlsanity.com/paper/2312.14235/) (2023)
- [AONeuS: A Neural Rendering Framework for Acoustic-Optical Sensor Fusion](https://mlsanity.com/paper/2402.03309/) (2024)
- [Rapid Network Adaptation: Learning to Adapt Neural Networks Using Test-Time Feedback](https://mlsanity.com/paper/2309.15762/) (2023)
- [Shakes on a Plane: Unsupervised Depth Estimation from Unstabilized Photography](https://mlsanity.com/paper/2212.12324/) (2022)
- [Instantaneous Perception of Moving Objects in 3D](https://mlsanity.com/paper/2405.02781/) (2024)
- [Depth-Aware Image Compositing Model for Parallax Camera Motion Blur](https://mlsanity.com/paper/2303.09334/) (2023)
- [Background Prompting for Improved Object Depth](https://mlsanity.com/paper/2306.05428/) (2023)
- [Iterative approach to reconstructing neural disparity fields from light-field data](https://mlsanity.com/paper/2407.15380/) (2024)
- [Fine Dense Alignment of Image Bursts through Camera Pose and Depth Estimation](https://mlsanity.com/paper/2312.05190/) (2023)
- [Mining Attribute Subspaces for Efficient Fine-tuning of 3D Foundation Models](https://mlsanity.com/paper/2604.10095/) (2026)

## Abstract (arXiv)

Modern smartphones can continuously stream multi-megapixel RGB images at 60Hz, synchronized with high-quality 3D pose information and low-resolution LiDAR-driven depth estimates. During a snapshot photograph, the natural unsteadiness of the photographer's hands offers millimeter-scale variation in camera pose, which we can capture along with RGB and depth in a circular buffer. In this work we explore how, from a bundle of these measurements acquired during viewfinding, we can combine dense micro-baseline parallax cues with kilopixel LiDAR depth to distill a high-fidelity depth map. We take a test-time optimization approach and train a coordinate MLP to output photometrically and geometrically consistent depth estimates at the continuous coordinates along the path traced by the photographer's natural hand shake. With no additional hardware, artificial hand motion, or user interaction beyond the press of a button, our proposed method brings high-resolution depth estimates to point-and-shoot "tabletop" photography -- textured objects at close range.