# Fine Dense Alignment of Image Bursts through Camera Pose and Depth Estimation

arXiv: https://arxiv.org/abs/2312.05190 · HTML: https://mlsanity.com/paper/2312.05190/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 39 references that are in the map.
- First posted on arXiv 8 Dec 2023.

## Builds on (most cited first, 18 of 18)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [Robust Consistent Video Depth Estimation](https://mlsanity.com/paper/2012.05901/) (2020)
- [A micro Lie theory for state estimation in robotics](https://mlsanity.com/paper/1812.01537/) (2018)
- [Handheld Multi-Frame Super-Resolution](https://mlsanity.com/paper/1905.03277/) (2019)
- [Deep Burst Super-Resolution](https://mlsanity.com/paper/2101.10997/) (2021)
- [Mobile Computational Photography: A Tour](https://mlsanity.com/paper/2102.09000/) (2021)
- [Handheld Mobile Photography in Very Low Light](https://mlsanity.com/paper/1910.11336/) (2019)
- [Deep Reparametrization of Multi-Frame Super-Resolution and Denoising](https://mlsanity.com/paper/2108.08286/) (2021)
- [BSRT: Improving Burst Super-Resolution with Swin Transformer and Flow-Guided Deformable Alignment](https://mlsanity.com/paper/2204.08332/) (2022)
- [Lucas-Kanade Reloaded: End-to-End Super-Resolution from Raw Image Bursts](https://mlsanity.com/paper/2104.06191/) (2021)
- [High Dynamic Range and Super-Resolution from Raw Image Bursts](https://mlsanity.com/paper/2207.14671/) (2022)
- [The Implicit Values of A Good Hand Shake: Handheld Multi-Frame Neural Depth Refinement](https://mlsanity.com/paper/2111.13738/) (2021)
- [Gated Multi-Resolution Transfer Network for Burst Restoration and Enhancement](https://mlsanity.com/paper/2304.06703/) (2023)
- [Shakes on a Plane: Unsupervised Depth Estimation from Unstabilized Photography](https://mlsanity.com/paper/2212.12324/) (2022)
- [Fundamental Limits in Multi-image Alignment](https://mlsanity.com/paper/1602.01541/) (2016)
- [A Practical Guide to Multi-image Alignment](https://mlsanity.com/paper/1802.03280/) (2018)

## Abstract (arXiv)

This paper introduces a novel approach to the fine alignment of images in a burst captured by a handheld camera. In contrast to traditional techniques that estimate two-dimensional transformations between frame pairs or rely on discrete correspondences, the proposed algorithm establishes dense correspondences by optimizing both the camera motion and surface depth and orientation at every pixel. This approach improves alignment, particularly in scenarios with parallax challenges. Extensive experiments with synthetic bursts featuring small and even tiny baselines demonstrate that it outperforms the best optical flow methods available today in this setting, without requiring any training. Beyond enhanced alignment, our method opens avenues for tasks beyond simple image restoration, such as depth estimation and 3D reconstruction, as supported by promising preliminary results. This positions our approach as a versatile tool for various burst image processing applications.