# GenNBV: Generalizable Next-Best-View Policy for Active 3D Reconstruction

arXiv: https://arxiv.org/abs/2402.16174 · HTML: https://mlsanity.com/paper/2402.16174/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 55 references that are in the map.
- First posted on arXiv 25 Feb 2024.

## Builds on (most cited first, 30 of 40)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Objaverse: A Universe of Annotated 3D Objects](https://mlsanity.com/paper/2212.08051/) (2022)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [TensoRF: Tensorial Radiance Fields](https://mlsanity.com/paper/2203.09517/) (2022)
- [Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction](https://mlsanity.com/paper/2111.11215/) (2021)
- [The Replica Dataset: A Digital Replica of Indoor Spaces](https://mlsanity.com/paper/1906.05797/) (2019)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [CURL: Contrastive Unsupervised Representations for Reinforcement Learning](https://mlsanity.com/paper/2004.04136/) (2020)
- [Open3D: A Modern Library for 3D Data Processing](https://mlsanity.com/paper/1801.09847/) (2018)
- [NICE-SLAM: Neural Implicit Scalable Encoding for SLAM](https://mlsanity.com/paper/2112.12130/) (2021)
- [Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning](https://mlsanity.com/paper/2109.11978/) (2021)
- [Leveraging Procedural Generation to Benchmark Reinforcement Learning](https://mlsanity.com/paper/1912.01588/) (2019)
- [iMAP: Implicit Mapping and Positioning in Real-Time](https://mlsanity.com/paper/2103.12352/) (2021)
- [OmniObject3D: Large-Vocabulary 3D Object Dataset for Realistic Perception, Reconstruction and Generation](https://mlsanity.com/paper/2301.07525/) (2023)
- [Learning to Explore using Active Neural SLAM](https://mlsanity.com/paper/2004.05155/) (2020)
- [MetaDrive: Composing Diverse Driving Scenarios for Generalizable Reinforcement Learning](https://mlsanity.com/paper/2109.12674/) (2021)
- [NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Video](https://mlsanity.com/paper/2104.00681/) (2021)
- [BungeeNeRF: Progressive Neural Radiance Field for Extreme Multi-scale Scene Rendering](https://mlsanity.com/paper/2112.05504/) (2021)
- [NeRF-SLAM: Real-Time Dense Monocular SLAM with Neural Radiance Fields](https://mlsanity.com/paper/2210.13641/) (2022)
- [MatrixCity: A Large-scale City Dataset for City-scale Neural Rendering and Beyond](https://mlsanity.com/paper/2309.16553/) (2023)
- [ESLAM: Efficient Dense SLAM System Based on Hybrid Representation of Signed Distance Fields](https://mlsanity.com/paper/2211.11704/) (2022)
- [Vision-Only Robot Navigation in a Neural Radiance World](https://mlsanity.com/paper/2110.00168/) (2021)
- [ActiveNeRF: Learning where to See with Uncertainty Estimation](https://mlsanity.com/paper/2209.08546/) (2022)
- [Learning Exploration Policies for Navigation](https://mlsanity.com/paper/1903.01959/) (2019)
- [iSDF: Real-Time Neural Signed Distance Fields for Robot Perception](https://mlsanity.com/paper/2204.02296/) (2022)
- [NeRFusion: Fusing Radiance Fields for Large-Scale Scene Reconstruction](https://mlsanity.com/paper/2203.11283/) (2022)
- [Neural 3D Reconstruction in the Wild](https://mlsanity.com/paper/2205.12955/) (2022)
- [ScenarioNet: Open-Source Platform for Large-Scale Traffic Scenario Simulation and Modeling](https://mlsanity.com/paper/2306.12241/) (2023)

## Built on by (most cited first, 30 of 39)

- [NextBestPath: Efficient 3D Mapping of Unseen Environments](https://mlsanity.com/paper/2502.05378/) (2025)
- [GLEAM: Learning Generalizable Exploration Policy for Active Mapping in Complex 3D Indoor Scenes](https://mlsanity.com/paper/2505.20294/) (2025)
- [Observe Then Act: Asynchronous Active Vision-Action Model for Robotic Manipulation](https://mlsanity.com/paper/2409.14891/) (2024)
- [Auto3R: Automated 3D Reconstruction and Scanning via Data-driven Uncertainty Quantification](https://mlsanity.com/paper/2512.04528/) (2025)
- [AREA3D: Active Reconstruction Agent with Unified Feed-Forward 3D Perception and Vision-Language Guidance](https://mlsanity.com/paper/2512.05131/) (2025)
- [VIN-NBV: A View Introspection Network for Next-Best-View Selection](https://mlsanity.com/paper/2505.06219/) (2025)
- [ACT-R: Adaptive Camera Trajectories for Single View 3D Reconstruction](https://mlsanity.com/paper/2505.08239/) (2025)
- [SA-ResGS: Self-Augmented Residual 3D Gaussian Splatting for Next Best View Selection](https://mlsanity.com/paper/2601.03024/) (2026)
- [ObjSplat: Geometry-Aware Gaussian Surfels for Active Object Reconstruction](https://mlsanity.com/paper/2601.06997/) (2026)
- [DAV-GSWT: Diffusion-Active-View Sampling for Data-Efficient Gaussian Splatting Wang Tiles](https://mlsanity.com/paper/2602.15355/) (2026)
- [Active View Selection with Perturbed Gaussian Ensemble for Tomographic Reconstruction](https://mlsanity.com/paper/2603.06852/) (2026)
- [OA-NBV: Occlusion-Aware Next-Best-View Planning for Human-Centered Active Perception on Mobile Robots](https://mlsanity.com/paper/2603.11072/) (2026)
- [Hestia: Voxel-Face-Aware Hierarchical Next-Best-View Acquisition for Efficient 3D Reconstruction](https://mlsanity.com/paper/2508.01014/) (2025)
- [CoRe-GS: Coarse-to-Refined Gaussian Splatting with Semantic Object Focus](https://mlsanity.com/paper/2509.04859/) (2025)
- [Aesthetic Camera Viewpoint Suggestion with 3D Aesthetic Field](https://mlsanity.com/paper/2602.20363/) (2026)
- [CM-EVS: Sparse Panoramic RGB-D-Pose Data for Complete Scene Coverage](https://mlsanity.com/paper/2605.15597/) (2026)
- [Diversity-aware View Partitioning for Scalable VGGT](https://mlsanity.com/paper/2607.01885/) (2026)
- [GO-PRE: Goal-Oriented Next-Best-View Selection via Predictive Rendering Entropy for Active 3D Reconstruction](https://mlsanity.com/paper/2607.29037/) (2026)
- [Boundary Exploration of Next Best View Policy in 3D Robotic Scanning](https://mlsanity.com/paper/2412.10444/) (2024)
- [Preference-Driven Active 3D Scene Representation for Robotic Inspection in Nuclear Decommissioning](https://mlsanity.com/paper/2504.02161/) (2025)
- [Exploiting Radiance Fields for Grasp Generation on Novel Synthetic Views](https://mlsanity.com/paper/2505.11467/) (2025)
- [Task-Driven Implicit Representations for Automated Design of LiDAR Systems](https://mlsanity.com/paper/2505.22344/) (2025)
- [SCORP: Scene-Consistent Object Refinement via Proxy Generation and Tuning](https://mlsanity.com/paper/2506.23835/) (2025)
- [Camera Splatting for Continuous View Optimization](https://mlsanity.com/paper/2509.15677/) (2025)
- [SOAR: Regression-based LiDAR Relocalization for UAVs](https://mlsanity.com/paper/2602.13267/) (2026)
- [ScanDP: Generalizable 3D Scanning with Diffusion Policy](https://mlsanity.com/paper/2603.10390/) (2026)
- [FluidGaussian: Propagating Simulation-Based Uncertainty Toward Functionally-Intelligent 3D Reconstruction](https://mlsanity.com/paper/2603.21356/) (2026)
- [SonoSelect: Efficient Ultrasound Perception via Active Probe Exploration](https://mlsanity.com/paper/2604.05933/) (2026)
- [Remember to be Curious: Episodic Context and Persistent Worlds for 3D Exploration](https://mlsanity.com/paper/2605.22814/) (2026)
- [PInVerify: An Offline Embodied Benchmark for Active Instance Verification](https://mlsanity.com/paper/2605.30639/) (2026)

## Abstract (arXiv)

While recent advances in neural radiance field enable realistic digitization for large-scale scenes, the image-capturing process is still time-consuming and labor-intensive. Previous works attempt to automate this process using the Next-Best-View (NBV) policy for active 3D reconstruction. However, the existing NBV policies heavily rely on hand-crafted criteria, limited action space, or per-scene optimized representations. These constraints limit their cross-dataset generalizability. To overcome them, we propose GenNBV, an end-to-end generalizable NBV policy. Our policy adopts a reinforcement learning (RL)-based framework and extends typical limited action space to 5D free space. It empowers our agent drone to scan from any viewpoint, and even interact with unseen geometries during training. To boost the cross-dataset generalizability, we also propose a novel multi-source state embedding, including geometric, semantic, and action representations. We establish a benchmark using the Isaac Gym simulator with the Houses3K and OmniObject3D datasets to evaluate this NBV policy. Experiments demonstrate that our policy achieves a 98.26% and 97.12% coverage ratio on unseen building-scale objects from these datasets, respectively, outperforming prior solutions.