# Neural Pixel Composition: 3D-4D View Synthesis from Multi-Views

arXiv: https://arxiv.org/abs/2207.10663 · HTML: https://mlsanity.com/paper/2207.10663/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 49 references that are in the map.
- First posted on arXiv 21 Jul 2022.

## Builds on (most cited first, 26 of 26)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [D-NeRF: Neural Radiance Fields for Dynamic Scenes](https://mlsanity.com/paper/2011.13961/) (2020)
- [Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations](https://mlsanity.com/paper/1906.01618/) (2019)
- [NeRF++: Analyzing and Improving Neural Radiance Fields](https://mlsanity.com/paper/2010.07492/) (2020)
- [Depth-supervised NeRF: Fewer Views and Faster Training for Free](https://mlsanity.com/paper/2107.02791/) (2021)
- [IBRNet: Learning Multi-View Image-Based Rendering](https://mlsanity.com/paper/2102.13090/) (2021)
- [Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines](https://mlsanity.com/paper/1905.00889/) (2019)
- [BARF: Bundle-Adjusting Neural Radiance Fields](https://mlsanity.com/paper/2104.06405/) (2021)
- [Advances in Neural Rendering](https://mlsanity.com/paper/2111.05849/) (2021)
- [State of the Art on Neural Rendering](https://mlsanity.com/paper/2004.03805/) (2020)
- [DeepMVS: Learning Multi-view Stereopsis](https://mlsanity.com/paper/1804.00650/) (2018)
- [Mixture of Volumetric Primitives for Efficient Neural Rendering](https://mlsanity.com/paper/2103.01954/) (2021)
- [Neural Radiance Flow for 4D View Synthesis and Video Processing](https://mlsanity.com/paper/2012.09790/) (2020)
- [Free View Synthesis](https://mlsanity.com/paper/2008.05511/) (2020)
- [NeX: Real-time View Synthesis with Neural Basis Expansion](https://mlsanity.com/paper/2103.05606/) (2021)
- [Self-Calibrating Neural Radiance Fields](https://mlsanity.com/paper/2108.13826/) (2021)
- [Stereo Radiance Fields (SRF): Learning View Synthesis for Sparse Views of Novel Scenes](https://mlsanity.com/paper/2104.06935/) (2021)
- [Stable View Synthesis](https://mlsanity.com/paper/2011.07233/) (2020)
- [Hierarchical Deep Stereo Matching on High-resolution Images](https://mlsanity.com/paper/1912.06704/) (2019)
- [KeypointNeRF: Generalizing Image-based Volumetric Avatars using Relative Spatial Encoding of Keypoints](https://mlsanity.com/paper/2205.04992/) (2022)
- [4D Visualization of Dynamic Events from Unconstrained Multi-View Videos](https://mlsanity.com/paper/2005.13532/) (2020)
- [DIVeR: Real-time and Accurate Neural Radiance Fields with Deterministic Integration for Volume Rendering](https://mlsanity.com/paper/2111.10427/) (2021)

## Built on by (most cited first, 7 of 7)

- [Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis](https://mlsanity.com/paper/2308.09713/) (2023)
- [SWinGS: Sliding Windows for Dynamic 3D Gaussian Splatting](https://mlsanity.com/paper/2312.13308/) (2023)
- [EgoGaussian: Dynamic Scene Understanding from Egocentric Video with 3D Gaussian Splatting](https://mlsanity.com/paper/2406.19811/) (2024)
- [Fast View Synthesis of Casual Videos with Soup-of-Planes](https://mlsanity.com/paper/2312.02135/) (2023)
- [VQA-Diff: Exploiting VQA and Diffusion for Zero-Shot Image-to-3D Vehicle Asset Generation in Autonomous Driving](https://mlsanity.com/paper/2407.06516/) (2024)
- [PAPR in Motion: Seamless Point-level 3D Scene Interpolation](https://mlsanity.com/paper/2406.05533/) (2024)
- [3D Gaussian Splatting against Moving Objects for High-Fidelity Street Scene Reconstruction](https://mlsanity.com/paper/2503.12001/) (2025)

## Abstract (arXiv)

We present Neural Pixel Composition (NPC), a novel approach for continuous 3D-4D view synthesis given only a discrete set of multi-view observations as input. Existing state-of-the-art approaches require dense multi-view supervision and an extensive computational budget. The proposed formulation reliably operates on sparse and wide-baseline multi-view imagery and can be trained efficiently within a few seconds to 10 minutes for hi-res (12MP) content, i.e., 200-400X faster convergence than existing methods. Crucial to our approach are two core novelties: 1) a representation of a pixel that contains color and depth information accumulated from multi-views for a particular location and time along a line of sight, and 2) a multi-layer perceptron (MLP) that enables the composition of this rich information provided for a pixel location to obtain the final color output. We experiment with a large variety of multi-view sequences, compare to existing approaches, and achieve better results in diverse and challenging settings. Finally, our approach enables dense 3D reconstruction from sparse multi-views, where COLMAP, a state-of-the-art 3D reconstruction approach, struggles.