# Real-Time Neural Rasterization for Large Scenes

arXiv: https://arxiv.org/abs/2311.05607 · HTML: https://mlsanity.com/paper/2311.05607/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Builds on 52 of its 73 references that are in the map.
- First posted on arXiv 9 Nov 2023.

## Builds on (most cited first, 30 of 52)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields](https://mlsanity.com/paper/2111.12077/) (2021)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [Plenoxels: Radiance Fields without Neural Networks](https://mlsanity.com/paper/2112.05131/) (2021)
- [TensoRF: Tensorial Radiance Fields](https://mlsanity.com/paper/2203.09517/) (2022)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction](https://mlsanity.com/paper/2111.11215/) (2021)
- [Neural Sparse Voxel Fields](https://mlsanity.com/paper/2007.11571/) (2020)
- [PlenOctrees for Real-time Rendering of Neural Radiance Fields](https://mlsanity.com/paper/2103.14024/) (2021)
- [Block-NeRF: Scalable Large Scene Neural View Synthesis](https://mlsanity.com/paper/2202.05263/) (2022)
- [Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision](https://mlsanity.com/paper/1912.07372/) (2019)
- [KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs](https://mlsanity.com/paper/2103.13744/) (2021)
- [Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines](https://mlsanity.com/paper/1905.00889/) (2019)
- [KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D](https://mlsanity.com/paper/2109.13410/) (2021)
- [FastNeRF: High-Fidelity Neural Rendering at 200FPS](https://mlsanity.com/paper/2103.10380/) (2021)
- [Habitat 2.0: Training Home Assistants to Rearrange their Habitat](https://mlsanity.com/paper/2106.14405/) (2021)
- [BlendedMVS: A Large-scale Dataset for Generalized Multi-view Stereo Networks](https://mlsanity.com/paper/1911.10127/) (2019)
- [MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction](https://mlsanity.com/paper/2206.00665/) (2022)
- [Baking Neural Radiance Fields for Real-Time View Synthesis](https://mlsanity.com/paper/2103.14645/) (2021)
- [Mega-NeRF: Scalable Construction of Large-Scale NeRFs for Virtual Fly-Throughs](https://mlsanity.com/paper/2112.10703/) (2021)
- [Neural Point-Based Graphics](https://mlsanity.com/paper/1906.08240/) (2019)
- [MobileNeRF: Exploiting the Polygon Rasterization Pipeline for Efficient Neural Field Rendering on Mobile Architectures](https://mlsanity.com/paper/2208.00277/) (2022)
- [UniSim: A Neural Closed-Loop Sensor Simulator](https://mlsanity.com/paper/2308.01898/) (2023)
- [Mixture of Volumetric Primitives for Efficient Neural Rendering](https://mlsanity.com/paper/2103.01954/) (2021)
- [Urban Radiance Fields](https://mlsanity.com/paper/2111.14643/) (2021)

## Built on by (most cited first, 30 of 37)

- [A Hierarchical 3D Gaussian Representation for Real-Time Rendering of Very Large Datasets](https://mlsanity.com/paper/2406.12080/) (2024)
- [textitS3Gaussian: Self-Supervised Street Gaussians for Autonomous Driving](https://mlsanity.com/paper/2405.20323/) (2024)
- [SMERF: Streamable Memory Efficient Radiance Fields for Real-Time Large-Scene Exploration](https://mlsanity.com/paper/2312.07541/) (2023)
- [SGD: Street View Synthesis with Gaussian Splatting and Diffusion Prior](https://mlsanity.com/paper/2403.20079/) (2024)
- [Dynamic 3D Gaussian Fields for Urban Areas](https://mlsanity.com/paper/2406.03175/) (2024)
- [G3R: Gradient Guided Generalizable Reconstruction](https://mlsanity.com/paper/2409.19405/) (2024)
- [Horizon-GS: Unified 3D Gaussian Splatting for Large-Scale Aerial-to-Ground Scenes](https://mlsanity.com/paper/2412.01745/) (2024)
- [FreeSim: Toward Free-viewpoint Camera Simulation in Driving Scenes](https://mlsanity.com/paper/2412.03566/) (2024)
- [DRAWER: Digital Reconstruction and Articulation With Environment Realism](https://mlsanity.com/paper/2504.15278/) (2025)
- [Multi-Level Neural Scene Graphs for Dynamic Urban Environments](https://mlsanity.com/paper/2404.00168/) (2024)
- [Video2Game: Real-time, Interactive, Realistic and Browser-Compatible Environment from a Single Video](https://mlsanity.com/paper/2404.09833/) (2024)
- [CoDa-4DGS: Dynamic Gaussian Splatting with Context and Deformation Awareness for Autonomous Driving](https://mlsanity.com/paper/2503.06744/) (2025)
- [MeshGS: Adaptive Mesh-Aligned Gaussian Splatting for High-Quality Rendering](https://mlsanity.com/paper/2410.08941/) (2024)
- [Radar Fields: Frequency-Space Neural Scene Representations for FMCW Radar](https://mlsanity.com/paper/2405.04662/) (2024)
- [Urban Architect: Steerable 3D Urban Scene Generation with Layout Prior](https://mlsanity.com/paper/2404.06780/) (2024)
- [GGS: Generalizable Gaussian Splatting for Lane Switching in Autonomous Driving](https://mlsanity.com/paper/2409.02382/) (2024)
- [Hier-SLAM: Scaling-up Semantics in SLAM with a Hierarchically Categorical Gaussian Splatting](https://mlsanity.com/paper/2409.12518/) (2024)
- [LidaRF: Delving into Lidar for Neural Radiance Field on Street Scenes](https://mlsanity.com/paper/2405.00900/) (2024)
- [HoloScene: Simulation-Ready Interactive 3D Worlds from a Single Video](https://mlsanity.com/paper/2510.05560/) (2025)
- [EMD: Explicit Motion Modeling for High-Quality Street Gaussian Splatting](https://mlsanity.com/paper/2411.15582/) (2024)
- [HGS-Mapping: Online Dense Mapping Using Hybrid Gaussian Representation in Urban Scenes](https://mlsanity.com/paper/2403.20159/) (2024)
- [TCLC-GS: Tightly Coupled LiDAR-Camera Gaussian Splatting for Autonomous Driving](https://mlsanity.com/paper/2404.02410/) (2024)
- [Dynamic Scene Reconstruction: Recent Advance in Real-time Rendering and Streaming](https://mlsanity.com/paper/2503.08166/) (2025)
- [Flux4D: Flow-based Unsupervised 4D Reconstruction](https://mlsanity.com/paper/2512.03210/) (2025)
- [UrbanCAD: Towards Highly Controllable and Photorealistic 3D Vehicles for Urban Scene Simulation](https://mlsanity.com/paper/2411.19292/) (2024)
- [XScale-NVS: Cross-Scale Novel View Synthesis with Hash Featurized Manifold](https://mlsanity.com/paper/2403.19517/) (2024)
- [FlexDrive: Toward Trajectory Flexibility in Driving Scene Reconstruction and Rendering](https://mlsanity.com/paper/2502.21093/) (2025)
- [Gated Fields: Learning Scene Reconstruction from Gated Videos](https://mlsanity.com/paper/2405.19819/) (2024)
- [IM360: Large-scale Indoor Mapping with 360 Cameras](https://mlsanity.com/paper/2502.12545/) (2025)
- [FreeDriveRF: Monocular RGB Dynamic NeRF without Poses for Autonomous Driving via Point-Level Dynamic-Static Decoupling](https://mlsanity.com/paper/2505.09406/) (2025)

## Abstract (arXiv)

We propose a new method for realistic real-time novel-view synthesis (NVS) of large scenes. Existing neural rendering methods generate realistic results, but primarily work for small scale scenes (<50 square meters) and have difficulty at large scale (>10000 square meters). Traditional graphics-based rasterization rendering is fast for large scenes but lacks realism and requires expensive manually created assets. Our approach combines the best of both worlds by taking a moderate-quality scaffold mesh as input and learning a neural texture field and shader to model view-dependant effects to enhance realism, while still using the standard graphics pipeline for real-time rendering. Our method outperforms existing neural rendering methods, providing at least 30x faster rendering with comparable or better realism for large self-driving and drone scenes. Our work is the first to enable real-time rendering of large real-world scenes.