# Single-View View Synthesis in the Wild with Learned Adaptive Multiplane Images

arXiv: https://arxiv.org/abs/2205.11733 · HTML: https://mlsanity.com/paper/2205.11733/

## Key facts

- Cited by 59 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 51 references that are in the map.
- First posted on arXiv 24 May 2022.

## Builds on (most cited first, 30 of 41)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer](https://mlsanity.com/paper/1907.01341/) (2019)
- [Neural Sparse Voxel Fields](https://mlsanity.com/paper/2007.11571/) (2020)
- [COCO-Stuff: Thing and Stuff Classes in Context](https://mlsanity.com/paper/1612.03716/) (2016)
- [IBRNet: Learning Multi-View Image-Based Rendering](https://mlsanity.com/paper/2102.13090/) (2021)
- [Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines](https://mlsanity.com/paper/1905.00889/) (2019)
- [Free-Form Image Inpainting with Gated Convolution](https://mlsanity.com/paper/1806.03589/) (2018)
- [TartanAir: A Dataset to Push the Limits of Visual SLAM](https://mlsanity.com/paper/2003.14338/) (2020)
- [SynSin: End-to-end View Synthesis from a Single Image](https://mlsanity.com/paper/1912.08804/) (2019)
- [Neural Point-Based Graphics](https://mlsanity.com/paper/1906.08240/) (2019)
- [Single-View View Synthesis with Multiplane Images](https://mlsanity.com/paper/2004.11364/) (2020)
- [DeepView: View Synthesis with Learned Gradient Descent](https://mlsanity.com/paper/1906.07316/) (2019)
- [Photographic Image Synthesis with Cascaded Refinement Networks](https://mlsanity.com/paper/1707.09405/) (2017)
- [Stereo Magnification: Learning View Synthesis using Multiplane Images](https://mlsanity.com/paper/1805.09817/) (2018)
- [3D Photography using Context-aware Layered Depth Inpainting](https://mlsanity.com/paper/2004.04727/) (2020)
- [Focal Frequency Loss for Image Reconstruction and Synthesis](https://mlsanity.com/paper/2012.12821/) (2020)
- [Pushing the Boundaries of View Extrapolation with Multiplane Images](https://mlsanity.com/paper/1905.00413/) (2019)
- [Neural Volumes: Learning Dynamic Renderable Volumes from Images](https://mlsanity.com/paper/1906.07751/) (2019)
- [Stable View Synthesis](https://mlsanity.com/paper/2011.07233/) (2020)
- [MINE: Towards Continuous Depth MPI with NeRF for Novel View Synthesis](https://mlsanity.com/paper/2103.14910/) (2021)
- [3D Ken Burns Effect from a Single Image](https://mlsanity.com/paper/1909.05483/) (2019)
- [DPSNet: End-to-end Deep Plane Sweep Stereo](https://mlsanity.com/paper/1905.00538/) (2019)
- [Neural Lumigraph Rendering](https://mlsanity.com/paper/2103.11571/) (2021)
- [Geometry-Free View Synthesis: Transformers and no 3D Priors](https://mlsanity.com/paper/2104.07652/) (2021)

## Built on by (most cited first, 30 of 59)

- [ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis](https://mlsanity.com/paper/2409.02048/) (2024)
- [Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior](https://mlsanity.com/paper/2303.14184/) (2023)
- [3D-aware Image Generation using 2D Diffusion Models](https://mlsanity.com/paper/2303.17905/) (2023)
- [ViP-NeRF: Visibility Prior for Sparse Input Neural Radiance Fields](https://mlsanity.com/paper/2305.00041/) (2023)
- [MonoNeRD: NeRF-like Representations for Monocular 3D Object Detection](https://mlsanity.com/paper/2308.09421/) (2023)
- [Pseudo-Generalized Dynamic View Synthesis from a Video](https://mlsanity.com/paper/2310.08587/) (2023)
- [3DitScene: Editing Any Scene via Language-guided Disentangled Gaussian Splatting](https://mlsanity.com/paper/2405.18424/) (2024)
- [3D Cinemagraphy from a Single Image](https://mlsanity.com/paper/2303.05724/) (2023)
- [Fast View Synthesis of Casual Videos with Soup-of-Planes](https://mlsanity.com/paper/2312.02135/) (2023)
- [SVG: 3D Stereoscopic Video Generation via Denoising Frame Matrix](https://mlsanity.com/paper/2407.00367/) (2024)
- [Positional Encoding Field](https://mlsanity.com/paper/2510.20385/) (2025)
- [Sharp Monocular View Synthesis in Less Than a Second](https://mlsanity.com/paper/2512.10685/) (2025)
- [Unaligned RGB Guided Hyperspectral Image Super-Resolution with Spatial-Spectral Concordance](https://mlsanity.com/paper/2505.02109/) (2025)
- [FrugalNeRF: Fast Convergence for Extreme Few-shot Novel View Synthesis without Learned Priors](https://mlsanity.com/paper/2410.16271/) (2024)
- [Pano2Room: Novel View Synthesis from a Single Indoor Panorama](https://mlsanity.com/paper/2408.11413/) (2024)
- [Tiled Multiplane Images for Practical 3D Photography](https://mlsanity.com/paper/2309.14291/) (2023)
- [Reconstruct, Inpaint, Test-Time Finetune: Dynamic Novel-view Synthesis from Monocular Videos](https://mlsanity.com/paper/2507.12646/) (2025)
- [Slice3D: Multi-Slice, Occlusion-Revealing, Single View 3D Reconstruction](https://mlsanity.com/paper/2312.02221/) (2023)
- [Structural Multiplane Image: Bridging Neural View Synthesis and 3D Reconstruction](https://mlsanity.com/paper/2303.05937/) (2023)
- [Diffusion Models are Geometry Critics: Single Image 3D Editing Using Pre-Trained Diffusion Priors](https://mlsanity.com/paper/2403.11503/) (2024)
- [SpatialDreamer: Self-supervised Stereo Video Synthesis from Monocular Input](https://mlsanity.com/paper/2411.11934/) (2024)
- [LiftImage3D: Lifting Any Single Image to 3D Gaussians with Video Generation Priors](https://mlsanity.com/paper/2412.09597/) (2024)
- [High-Fidelity Novel View Synthesis via Splatting-Guided Diffusion](https://mlsanity.com/paper/2502.12752/) (2025)
- [M2SVid: End-to-End Inpainting and Refinement for Monocular-to-Stereo Video Conversion](https://mlsanity.com/paper/2505.16565/) (2025)
- [Global Latent Neural Rendering](https://mlsanity.com/paper/2312.08338/) (2023)
- [Learning 3D Photography Videos via Self-supervised Diffusion on Single Images](https://mlsanity.com/paper/2302.10781/) (2023)
- [MPI-Flow: Learning Realistic Optical Flow with Multiplane Images](https://mlsanity.com/paper/2309.06714/) (2023)
- [SpatialMe: Stereo Video Conversion Using Depth-Warping and Blend-Inpainting](https://mlsanity.com/paper/2412.11512/) (2024)
- [Efficient View Synthesis and 3D-based Multi-Frame Denoising with Multiplane Feature Representations](https://mlsanity.com/paper/2303.18139/) (2023)
- [VaLID: Variable-Length Input Diffusion for Novel View Synthesis](https://mlsanity.com/paper/2312.08892/) (2023)

## Abstract (arXiv)

This paper deals with the challenging task of synthesizing novel views for in-the-wild photographs. Existing methods have shown promising results leveraging monocular depth estimation and color inpainting with layered depth representations. However, these methods still have limited capability to handle scenes with complex 3D geometry. We propose a new method based on the multiplane image (MPI) representation. To accommodate diverse scene layouts in the wild and tackle the difficulty in producing high-dimensional MPI contents, we design a network structure that consists of two novel modules, one for plane depth adjustment and another for depth-aware color prediction. The former adjusts the initial plane positions using the RGBD context feature and an attention mechanism. Given adjusted depth values, the latter predicts the color and density for each plane separately with proper inter-plane interactions achieved via a feature masking strategy. To train our method, we construct large-scale stereo training data using only unconstrained single-view image collections by a simple yet effective warp-back strategy. The experiments on both synthetic and real datasets demonstrate that our trained model works remarkably well and achieves state-of-the-art results.