# NeU-NBV: Next Best View Planning Using Uncertainty Estimation in Image-Based Neural Rendering

arXiv: https://arxiv.org/abs/2303.01284 · HTML: https://mlsanity.com/paper/2303.01284/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Mar 2023.

## Built on by (most cited first, 30 of 54)

- [ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis](https://mlsanity.com/paper/2409.02048/) (2024)
- [Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AI](https://mlsanity.com/paper/2407.06886/) (2024)
- [Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysis](https://mlsanity.com/paper/2312.08782/) (2023)
- [PUP 3D-GS: Principled Uncertainty Pruning for 3D Gaussian Splatting](https://mlsanity.com/paper/2406.10219/) (2024)
- [Bayes' Rays: Uncertainty Quantification for Neural Radiance Fields](https://mlsanity.com/paper/2309.03185/) (2023)
- [NeRF On-the-go: Exploiting Uncertainty for Distractor-free NeRFs in the Wild](https://mlsanity.com/paper/2405.18715/) (2024)
- [ActiveGS: Active Scene Reconstruction Using Gaussian Splatting](https://mlsanity.com/paper/2412.17769/) (2024)
- [Active Implicit Object Reconstruction using Uncertainty-guided Next-Best-View Optimization](https://mlsanity.com/paper/2303.16739/) (2023)
- [Neural Visibility Field for Uncertainty-Driven Active Mapping](https://mlsanity.com/paper/2406.06948/) (2024)
- [NeRF Director: Revisiting View Selection in Neural Volume Rendering](https://mlsanity.com/paper/2406.08839/) (2024)
- [VISO-Grasp: Vision-Language Informed Spatial Object-centric 6-DoF Active View Planning and Grasping in Clutter and Invisibility](https://mlsanity.com/paper/2503.12609/) (2025)
- [Sources of Uncertainty in 3D Scene Reconstruction](https://mlsanity.com/paper/2409.06407/) (2024)
- [Multi-view 3D Object Reconstruction and Uncertainty Modelling with Neural Shape Prior](https://mlsanity.com/paper/2306.11739/) (2023)
- [Neural Stochastic Screened Poisson Reconstruction](https://mlsanity.com/paper/2309.11993/) (2023)
- [These Magic Moments: Differentiable Uncertainty Quantification of Radiance Field Models](https://mlsanity.com/paper/2503.14665/) (2025)
- [ExploreGS: Explorable 3D Scene Reconstruction with Virtual Camera Samplings and Diffusion Priors](https://mlsanity.com/paper/2508.06014/) (2025)
- [SO-NeRF: Active View Planning for NeRF using Surrogate Objectives](https://mlsanity.com/paper/2312.03266/) (2023)
- [Uncertainty in Action: Confidence Elicitation in Embodied Agents](https://mlsanity.com/paper/2503.10628/) (2025)
- [Peering into the Unknown: Active View Selection with Neural Uncertainty Maps for 3D Reconstruction](https://mlsanity.com/paper/2506.14856/) (2025)
- [Uncertainty-driven 3D Gaussian Splatting Active Mapping via Anisotropic Visibility Field](https://mlsanity.com/paper/2605.30342/) (2026)
- [Exploiting Priors from 3D Diffusion Models for RGB-Based One-Shot View Planning](https://mlsanity.com/paper/2403.16803/) (2024)
- [Uncertainty-aware Active Learning of NeRF-based Object Models for Robot Manipulators using Visual and Re-orientation Actions](https://mlsanity.com/paper/2404.01812/) (2024)
- [Frequency-based View Selection in Gaussian Splatting Reconstruction](https://mlsanity.com/paper/2409.16470/) (2024)
- [Auto3R: Automated 3D Reconstruction and Scanning via Data-driven Uncertainty Quantification](https://mlsanity.com/paper/2512.04528/) (2025)
- [AREA3D: Active Reconstruction Agent with Unified Feed-Forward 3D Perception and Vision-Language Guidance](https://mlsanity.com/paper/2512.05131/) (2025)
- [Instant Uncertainty Calibration of NeRFs Using a Meta-Calibrator](https://mlsanity.com/paper/2312.02350/) (2023)
- [ProvNeRF: Modeling per Point Provenance in NeRFs as a Stochastic Field](https://mlsanity.com/paper/2401.08140/) (2024)
- [REF2-NeRF: Reflection and Refraction aware Neural Radiance Field](https://mlsanity.com/paper/2311.17116/) (2023)
- [DNRSelect: Active Best View Selection for Deferred Neural Rendering](https://mlsanity.com/paper/2501.12150/) (2025)
- [DM-OSVP++: One-Shot View Planning Using 3D Diffusion Models for Active RGB-Based Object Reconstruction](https://mlsanity.com/paper/2504.11674/) (2025)

## Abstract (arXiv)

Autonomous robotic tasks require actively perceiving the environment to achieve application-specific goals. In this paper, we address the problem of positioning an RGB camera to collect the most informative images to represent an unknown scene, given a limited measurement budget. We propose a novel mapless planning framework to iteratively plan the next best camera view based on collected image measurements. A key aspect of our approach is a new technique for uncertainty estimation in image-based neural rendering, which guides measurement acquisition at the most uncertain view among view candidates, thus maximising the information value during data collection. By incrementally adding new measurements into our image collection, our approach efficiently explores an unknown scene in a mapless manner. We show that our uncertainty estimation is generalisable and valuable for view planning in unknown scenes. Our planning experiments using synthetic and real-world data verify that our uncertainty-guided approach finds informative images leading to more accurate scene representations when compared against baselines.