# VET: Visual Error Tomography for Point Cloud Completion and High-Quality Neural Rendering

arXiv: https://arxiv.org/abs/2311.04634 · HTML: https://mlsanity.com/paper/2311.04634/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 58 references that are in the map.
- First posted on arXiv 8 Nov 2023.

## Builds on (most cited first, 30 of 31)

- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [Plenoxels: Radiance Fields without Neural Networks](https://mlsanity.com/paper/2112.05131/) (2021)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections](https://mlsanity.com/paper/2008.02268/) (2020)
- [NeRF++: Analyzing and Improving Neural Radiance Fields](https://mlsanity.com/paper/2010.07492/) (2020)
- [Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines](https://mlsanity.com/paper/1905.00889/) (2019)
- [Mega-NeRF: Scalable Construction of Large-Scale NeRFs for Virtual Fly-Throughs](https://mlsanity.com/paper/2112.10703/) (2021)
- [SynSin: End-to-end View Synthesis from a Single Image](https://mlsanity.com/paper/1912.08804/) (2019)
- [Neural Point-Based Graphics](https://mlsanity.com/paper/1906.08240/) (2019)
- [Single-View View Synthesis with Multiplane Images](https://mlsanity.com/paper/2004.11364/) (2020)
- [Deferred Neural Rendering: Image Synthesis using Neural Textures](https://mlsanity.com/paper/1904.12356/) (2019)
- [Free View Synthesis](https://mlsanity.com/paper/2008.05511/) (2020)
- [Stereo Magnification: Learning View Synthesis using Multiplane Images](https://mlsanity.com/paper/1805.09817/) (2018)
- [Pushing the Boundaries of View Extrapolation with Multiplane Images](https://mlsanity.com/paper/1905.00413/) (2019)
- [Point-Based Neural Rendering with Per-View Optimization](https://mlsanity.com/paper/2109.02369/) (2021)
- [DeepStereo: Learning to Predict New Views from the World's Imagery](https://mlsanity.com/paper/1506.06825/) (2015)
- [View Synthesis by Appearance Flow](https://mlsanity.com/paper/1605.03557/) (2016)
- [Stable View Synthesis](https://mlsanity.com/paper/2011.07233/) (2020)
- [Neural Rerendering in the Wild](https://mlsanity.com/paper/1904.04290/) (2019)
- [Differentiable Point-Based Radiance Fields for Efficient View Synthesis](https://mlsanity.com/paper/2205.14330/) (2022)
- [Neural Point Catacaustics for Novel-View Synthesis of Reflections](https://mlsanity.com/paper/2301.01087/) (2023)
- [Revealing Scenes by Inverting Structure from Motion Reconstructions](https://mlsanity.com/paper/1904.03303/) (2019)
- [A Large Dataset of Object Scans](https://mlsanity.com/paper/1602.02481/) (2016)
- [A Large-Scale Outdoor Multi-modal Dataset and Benchmark for Novel View Synthesis and Implicit Scene Reconstruction](https://mlsanity.com/paper/2301.06782/) (2023)
- [DeepWarp: Photorealistic Image Resynthesis for Gaze Manipulation](https://mlsanity.com/paper/1607.07215/) (2016)
- [View Synthesis with Sculpted Neural Points](https://mlsanity.com/paper/2205.05869/) (2022)
- [Software Rasterization of 2 Billion Points in Real Time](https://mlsanity.com/paper/2204.01287/) (2022)
- [NeAT: Neural Adaptive Tomography](https://mlsanity.com/paper/2202.02171/) (2022)

## Built on by (most cited first, 14 of 14)

- [RadSplat: Radiance Field-Informed Gaussian Splatting for Robust Real-Time Rendering with 900+ FPS](https://mlsanity.com/paper/2403.13806/) (2024)
- [Efficient Perspective-Correct 3D Gaussian Splatting Using Hybrid Transparency](https://mlsanity.com/paper/2410.08129/) (2024)
- [TRIPS: Trilinear Point Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2401.06003/) (2024)
- [VR-Splatting: Foveated Radiance Field Rendering via 3D Gaussian Splatting and Neural Points](https://mlsanity.com/paper/2410.17932/) (2024)
- [MirrorGaussian: Reflecting 3D Gaussians for Reconstructing Mirror Reflections](https://mlsanity.com/paper/2405.11921/) (2024)
- [EVolSplat: Efficient Volume-based Gaussian Splatting for Urban View Synthesis](https://mlsanity.com/paper/2503.20168/) (2025)
- [PEGASUS: Physically Enhanced Gaussian Splatting Simulation System for 6DoF Object Pose Dataset Generation](https://mlsanity.com/paper/2401.02281/) (2024)
- [INPC: Implicit Neural Point Clouds for Radiance Field Rendering](https://mlsanity.com/paper/2403.16862/) (2024)
- [Multi-Spectral Gaussian Splatting with Neural Color Representation](https://mlsanity.com/paper/2506.03407/) (2025)
- [Towards Integrating Multi-Spectral Imaging with Gaussian Splatting](https://mlsanity.com/paper/2509.00989/) (2025)
- [PEP-GS: Perceptually-Enhanced Precise Structured 3D Gaussians for View-Adaptive Rendering](https://mlsanity.com/paper/2411.05731/) (2024)
- [A Bag of Tricks for Efficient Implicit Neural Point Clouds](https://mlsanity.com/paper/2508.19140/) (2025)
- [SurfFill: Completion of LiDAR Point Clouds via Gaussian Surfel Splatting](https://mlsanity.com/paper/2512.03010/) (2025)
- [ES-Gaussian: Gaussian Splatting Mapping via Error Space-Based Gaussian Completion](https://mlsanity.com/paper/2410.06613/) (2024)

## Abstract (arXiv)

In the last few years, deep neural networks opened the doors for big advances in novel view synthesis. Many of these approaches are based on a (coarse) proxy geometry obtained by structure from motion algorithms. Small deficiencies in this proxy can be fixed by neural rendering, but larger holes or missing parts, as they commonly appear for thin structures or for glossy regions, still lead to distracting artifacts and temporal instability. In this paper, we present a novel neural-rendering-based approach to detect and fix such deficiencies. As a proxy, we use a point cloud, which allows us to easily remove outlier geometry and to fill in missing geometry without complicated topological operations. Keys to our approach are (i) a differentiable, blending point-based renderer that can blend out redundant points, as well as (ii) the concept of Visual Error Tomography (VET), which allows us to lift 2D error maps to identify 3D-regions lacking geometry and to spawn novel points accordingly. Furthermore, (iii) by adding points as nested environment maps, our approach allows us to generate high-quality renderings of the surroundings in the same pipeline. In our results, we show that our approach can improve the quality of a point cloud obtained by structure from motion and thus increase novel view synthesis quality significantly. In contrast to point growing techniques, the approach can also fix large-scale holes and missing thin structures effectively. Rendering quality outperforms state-of-the-art methods and temporal stability is significantly improved, while rendering is possible at real-time frame rates.