# Neural Spline Fields for Burst Image Fusion and Layer Separation

arXiv: https://arxiv.org/abs/2312.14235 · HTML: https://mlsanity.com/paper/2312.14235/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 73 references that are in the map.
- First posted on arXiv 21 Dec 2023.

## Builds on (most cited first, 30 of 43)

- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [PlenOctrees for Real-time Rendering of Neural Radiance Fields](https://mlsanity.com/paper/2103.14024/) (2021)
- [Resolution-robust Large Mask Inpainting with Fourier Convolutions](https://mlsanity.com/paper/2109.07161/) (2021)
- [Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fields](https://mlsanity.com/paper/2304.06706/) (2023)
- [Neuralangelo: High-Fidelity Neural Surface Reconstruction](https://mlsanity.com/paper/2306.03092/) (2023)
- [RAFT-Stereo: Multilevel Recurrent Field Transforms for Stereo Matching](https://mlsanity.com/paper/2109.07547/) (2021)
- [NeRF in the Dark: High Dynamic Range View Synthesis from Noisy Raw Images](https://mlsanity.com/paper/2111.13679/) (2021)
- [DynIBaR: Neural Dynamic Image-Based Rendering](https://mlsanity.com/paper/2211.11082/) (2022)
- [Burst Denoising with Kernel Prediction Networks](https://mlsanity.com/paper/1712.02327/) (2017)
- [Layered Neural Atlases for Consistent Video Editing](https://mlsanity.com/paper/2109.11418/) (2021)
- [RAFT-3D: Scene Flow using Rigid-Motion Embeddings](https://mlsanity.com/paper/2012.00726/) (2020)
- [NeRFReN: Neural Radiance Fields with Reflections](https://mlsanity.com/paper/2111.15234/) (2021)
- ["Double-DIP": Unsupervised Image Decomposition via Coupled Deep-Image-Priors](https://mlsanity.com/paper/1812.00467/) (2018)
- [Real-time Neural Radiance Caching for Path Tracing](https://mlsanity.com/paper/2106.12372/) (2021)
- [Foreground-aware Image Inpainting](https://mlsanity.com/paper/1901.05945/) (2019)
- [Handheld Multi-Frame Super-Resolution](https://mlsanity.com/paper/1905.03277/) (2019)
- [Deep Burst Super-Resolution](https://mlsanity.com/paper/2101.10997/) (2021)
- [Polarized Reflection Removal with Perfect Alignment in the Wild](https://mlsanity.com/paper/2003.12789/) (2020)
- [Mobile Computational Photography: A Tour](https://mlsanity.com/paper/2102.09000/) (2021)
- [Deformable Sprites for Unsupervised Video Decomposition](https://mlsanity.com/paper/2204.07151/) (2022)
- [Handheld Mobile Photography in Very Low Light](https://mlsanity.com/paper/1910.11336/) (2019)
- [Single Image Reflection Removal Exploiting Misaligned Training Data and Network Enhancements](https://mlsanity.com/paper/1904.00637/) (2019)
- [Omnimatte: Associating Objects and Their Effects in Video](https://mlsanity.com/paper/2105.06993/) (2021)
- [Deep Burst Denoising](https://mlsanity.com/paper/1712.05790/) (2017)
- [Explorable Super Resolution](https://mlsanity.com/paper/1912.01839/) (2019)

## Built on by (most cited first, 16 of 16)

- [DC-Gaussian: Improving 3D Gaussian Splatting for Reflective Dash Cam Videos](https://mlsanity.com/paper/2405.17705/) (2024)
- [AONeuS: A Neural Rendering Framework for Acoustic-Optical Sensor Fusion](https://mlsanity.com/paper/2402.03309/) (2024)
- [Flash-Split: 2D Reflection Removal with Flash Cues and Latent Diffusion Separation](https://mlsanity.com/paper/2501.00637/) (2024)
- [AdaGaR: Adaptive Gabor Representation for Dynamic Scene Reconstruction](https://mlsanity.com/paper/2601.00796/) (2026)
- [Neural Light Spheres for Implicit Image Stitching and View Synthesis](https://mlsanity.com/paper/2409.17924/) (2024)
- [FIRM: Flexible Interactive Reflection reMoval](https://mlsanity.com/paper/2406.01555/) (2024)
- [Flash-Splat: 3D Reflection Removal with Flash Cues and Gaussian Splats](https://mlsanity.com/paper/2410.02764/) (2024)
- [Instruct2See: Learning to Remove Any Obstructions Across Distributions](https://mlsanity.com/paper/2505.17649/) (2025)
- [Event2Audio: Event-Based Optical Vibration Sensing](https://mlsanity.com/paper/2507.03273/) (2025)
- [Spline Deformation Field](https://mlsanity.com/paper/2507.07521/) (2025)
- [Neural Atlas Graphs for Dynamic Scene Decomposition and Editing](https://mlsanity.com/paper/2509.16336/) (2025)
- [Dark3R: Learning Structure from Motion in the Dark](https://mlsanity.com/paper/2603.05330/) (2026)
- [SuperF: Neural Implicit Fields for Multi-Image Super-Resolution](https://mlsanity.com/paper/2512.09115/) (2025)
- [Deep Generative Adversarial Network for Occlusion Removal from a Single Image](https://mlsanity.com/paper/2409.13242/) (2024)
- [Under-Canopy Terrain Reconstruction in Dense Forests Using RGB Imaging and Neural 3D Reconstruction](https://mlsanity.com/paper/2601.22861/) (2026)
- [Reflection Separation from a Single Image via Joint Latent Diffusion](https://mlsanity.com/paper/2606.04107/) (2026)

## Abstract (arXiv)

Each photo in an image burst can be considered a sample of a complex 3D scene: the product of parallax, diffuse and specular materials, scene motion, and illuminant variation. While decomposing all of these effects from a stack of misaligned images is a highly ill-conditioned task, the conventional align-and-merge burst pipeline takes the other extreme: blending them into a single image. In this work, we propose a versatile intermediate representation: a two-layer alpha-composited image plus flow model constructed with neural spline fields -- networks trained to map input coordinates to spline control points. Our method is able to, during test-time optimization, jointly fuse a burst image capture into one high-resolution reconstruction and decompose it into transmission and obstruction layers. Then, by discarding the obstruction layer, we can perform a range of tasks including seeing through occlusions, reflection suppression, and shadow removal. Validated on complex synthetic and in-the-wild captures we find that, with no post-processing steps or learned priors, our generalizable model is able to outperform existing dedicated single-image and multi-view obstruction removal approaches.