# Shakes on a Plane: Unsupervised Depth Estimation from Unstabilized Photography

arXiv: https://arxiv.org/abs/2212.12324 · HTML: https://mlsanity.com/paper/2212.12324/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 87 references that are in the map.
- First posted on arXiv 22 Dec 2022.

## Builds on (most cited first, 30 of 44)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo](https://mlsanity.com/paper/2103.15595/) (2021)
- [Block-NeRF: Scalable Large Scene Neural View Synthesis](https://mlsanity.com/paper/2202.05263/) (2022)
- [BARF: Bundle-Adjusting Neural Radiance Fields](https://mlsanity.com/paper/2104.06405/) (2021)
- [AdaBins: Depth Estimation using Adaptive Bins](https://mlsanity.com/paper/2011.14141/) (2020)
- [Noise2Noise: Learning Image Restoration without Clean Data](https://mlsanity.com/paper/1803.04189/) (2018)
- [Pyramid Stereo Matching Network](https://mlsanity.com/paper/1803.08669/) (2018)
- [RAFT-Stereo: Multilevel Recurrent Field Transforms for Stereo Matching](https://mlsanity.com/paper/2109.07547/) (2021)
- [NeRF in the Dark: High Dynamic Range View Synthesis from Noisy Raw Images](https://mlsanity.com/paper/2111.13679/) (2021)
- [Unprocessing Images for Learned Raw Denoising](https://mlsanity.com/paper/1811.11127/) (2018)
- [Depth-Aware Video Frame Interpolation](https://mlsanity.com/paper/1904.00830/) (2019)
- [Consistent Video Depth Estimation](https://mlsanity.com/paper/2004.15021/) (2020)
- [DeMoN: Depth and Motion Network for Learning Monocular Stereo](https://mlsanity.com/paper/1612.02401/) (2016)
- [Monocular Dynamic View Synthesis: A Reality Check](https://mlsanity.com/paper/2210.13445/) (2022)
- [Robust Consistent Video Depth Estimation](https://mlsanity.com/paper/2012.05901/) (2020)
- [NeRF--: Neural Radiance Fields Without Known Camera Parameters](https://mlsanity.com/paper/2102.07064/) (2021)
- [Revisiting Single Image Depth Estimation: Toward Higher Resolution Maps with Accurate Object Boundaries](https://mlsanity.com/paper/1803.08673/) (2018)
- [Defocus Deblurring Using Dual-Pixel Data](https://mlsanity.com/paper/2005.00305/) (2020)
- [HITNet: Hierarchical Iterative Tile Refinement Network for Real-time Stereo Matching](https://mlsanity.com/paper/2007.12140/) (2020)
- [Boosting Monocular Depth Estimation Models to High-Resolution via Content-Adaptive Multi-Resolution Merging](https://mlsanity.com/paper/2105.14021/) (2021)
- [Real-time Neural Radiance Caching for Path Tracing](https://mlsanity.com/paper/2106.12372/) (2021)
- [Image-based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era](https://mlsanity.com/paper/1906.06543/) (2019)
- [Handheld Multi-Frame Super-Resolution](https://mlsanity.com/paper/1905.03277/) (2019)
- [Synthetic Depth-of-Field with a Single-Camera Mobile Phone](https://mlsanity.com/paper/1806.04171/) (2018)

## Built on by (most cited first, 8 of 8)

- [Neural Spline Fields for Burst Image Fusion and Layer Separation](https://mlsanity.com/paper/2312.14235/) (2023)
- [Neural Light Spheres for Implicit Image Stitching and View Synthesis](https://mlsanity.com/paper/2409.17924/) (2024)
- [Iterative approach to reconstructing neural disparity fields from light-field data](https://mlsanity.com/paper/2407.15380/) (2024)
- [Dark3R: Learning Structure from Motion in the Dark](https://mlsanity.com/paper/2603.05330/) (2026)
- [Solutions to Deepfakes: Can Camera Hardware, Cryptography, and Deep Learning Verify Real Images?](https://mlsanity.com/paper/2407.04169/) (2024)
- [Refinement of Monocular Depth Maps via Multi-View Differentiable Rendering](https://mlsanity.com/paper/2410.03861/) (2024)
- [Fine Dense Alignment of Image Bursts through Camera Pose and Depth Estimation](https://mlsanity.com/paper/2312.05190/) (2023)
- [Mining Attribute Subspaces for Efficient Fine-tuning of 3D Foundation Models](https://mlsanity.com/paper/2604.10095/) (2026)

## Abstract (arXiv)

Modern mobile burst photography pipelines capture and merge a short sequence of frames to recover an enhanced image, but often disregard the 3D nature of the scene they capture, treating pixel motion between images as a 2D aggregation problem. We show that in a ''long-burst'', forty-two 12-megapixel RAW frames captured in a two-second sequence, there is enough parallax information from natural hand tremor alone to recover high-quality scene depth. To this end, we devise a test-time optimization approach that fits a neural RGB-D representation to long-burst data and simultaneously estimates scene depth and camera motion. Our plane plus depth model is trained end-to-end, and performs coarse-to-fine refinement by controlling which multi-resolution volume features the network has access to at what time during training. We validate the method experimentally, and demonstrate geometrically accurate depth reconstructions with no additional hardware or separate data pre-processing and pose-estimation steps.