# LoLep: Single-View View Synthesis with Locally-Learned Planes and Self-Attention Occlusion Inference

arXiv: https://arxiv.org/abs/2307.12217 · HTML: https://mlsanity.com/paper/2307.12217/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 63 references that are in the map.
- First posted on arXiv 23 Jul 2023.

## Builds on (most cited first, 29 of 29)

- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections](https://mlsanity.com/paper/2008.02268/) (2020)
- [Nerfies: Deformable Neural Radiance Fields](https://mlsanity.com/paper/2011.12948/) (2020)
- [Unsupervised Monocular Depth Estimation with Left-Right Consistency](https://mlsanity.com/paper/1609.03677/) (2016)
- [AdaBins: Depth Estimation using Adaptive Bins](https://mlsanity.com/paper/2011.14141/) (2020)
- [DeepVoxels: Learning Persistent 3D Feature Embeddings](https://mlsanity.com/paper/1812.01024/) (2018)
- [SynSin: End-to-end View Synthesis from a Single Image](https://mlsanity.com/paper/1912.08804/) (2019)
- [Neural Point-Based Graphics](https://mlsanity.com/paper/1906.08240/) (2019)
- [Single-View View Synthesis with Multiplane Images](https://mlsanity.com/paper/2004.11364/) (2020)
- [3D Photography using Context-aware Layered Depth Inpainting](https://mlsanity.com/paper/2004.04727/) (2020)
- [Asymmetric Non-local Neural Networks for Semantic Segmentation](https://mlsanity.com/paper/1908.07678/) (2019)
- [SinNeRF: Training Neural Radiance Fields on Complex Scenes from a Single Image](https://mlsanity.com/paper/2204.00928/) (2022)
- [Evaluation of CNN-based Single-Image Depth Estimation Methods](https://mlsanity.com/paper/1805.01328/) (2018)
- [MINE: Towards Continuous Depth MPI with NeRF for Novel View Synthesis](https://mlsanity.com/paper/2103.14910/) (2021)
- [3D Ken Burns Effect from a Single Image](https://mlsanity.com/paper/1909.05483/) (2019)
- [Neural Rerendering in the Wild](https://mlsanity.com/paper/1904.04290/) (2019)
- [Extreme View Synthesis](https://mlsanity.com/paper/1812.04777/) (2018)
- [Layer-structured 3D Scene Inference via View Synthesis](https://mlsanity.com/paper/1807.10264/) (2018)
- [Deep3D: Fully Automatic 2D-to-3D Video Conversion with Deep Convolutional Neural Networks](https://mlsanity.com/paper/1604.03650/) (2016)
- [Monocular Depth Estimation Based On Deep Learning: An Overview](https://mlsanity.com/paper/2003.06620/) (2020)
- [Learning to Synthesize a 4D RGBD Light Field from a Single Image](https://mlsanity.com/paper/1708.03292/) (2017)
- [Single-View View Synthesis in the Wild with Learned Adaptive Multiplane Images](https://mlsanity.com/paper/2205.11733/) (2022)
- [NeRF in detail: Learning to sample for view synthesis](https://mlsanity.com/paper/2106.05264/) (2021)
- [Geometry-aware Deep Network for Single-Image Novel View Synthesis](https://mlsanity.com/paper/1804.06008/) (2018)
- [Neural Point-based Volumetric Avatar: Surface-guided Neural Points for Efficient and Photorealistic Volumetric Head Avatar](https://mlsanity.com/paper/2307.05000/) (2023)
- [MotionHint: Self-Supervised Monocular Visual Odometry with Motion Constraints](https://mlsanity.com/paper/2109.06768/) (2021)
- [ORBBuf: A Robust Buffering Method for Remote Visual SLAM](https://mlsanity.com/paper/2010.14861/) (2020)

## Built on by (most cited first, 2 of 2)

- [MeGA: Hybrid Mesh-Gaussian Head Avatar for High-Fidelity Rendering and Head Editing](https://mlsanity.com/paper/2404.19026/) (2024)
- [DecoRec: Decomposed 3D Scene Reconstruction from Single-View Images via Object-Level Diffusion](https://mlsanity.com/paper/2605.16807/) (2026)

## Abstract (arXiv)

We propose a novel method, LoLep, which regresses Locally-Learned planes from a single RGB image to represent scenes accurately, thus generating better novel views. Without the depth information, regressing appropriate plane locations is a challenging problem. To solve this issue, we pre-partition the disparity space into bins and design a disparity sampler to regress local offsets for multiple planes in each bin. However, only using such a sampler makes the network not convergent; we further propose two optimizing strategies that combine with different disparity distributions of datasets and propose an occlusion-aware reprojection loss as a simple yet effective geometric supervision technique. We also introduce a self-attention mechanism to improve occlusion inference and present a Block-Sampling Self-Attention (BS-SA) module to address the problem of applying self-attention to large feature maps. We demonstrate the effectiveness of our approach and generate state-of-the-art results on different datasets. Compared to MINE, our approach has an LPIPS reduction of 4.8%-9.0% and an RV reduction of 73.9%-83.5%. We also evaluate the performance on real-world images and demonstrate the benefits.