# UC-NeRF: Neural Radiance Field for Under-Calibrated Multi-view Cameras in Autonomous Driving

arXiv: https://arxiv.org/abs/2311.16945 · HTML: https://mlsanity.com/paper/2311.16945/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 47 references that are in the map.
- First posted on arXiv 28 Nov 2023.

## Builds on (most cited first, 30 of 38)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [Scalability in Perception for Autonomous Driving: Waymo Open Dataset](https://mlsanity.com/paper/1912.04838/) (2019)
- [Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields](https://mlsanity.com/paper/2111.12077/) (2021)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [SuperPoint: Self-Supervised Interest Point Detection and Description](https://mlsanity.com/paper/1712.07629/) (2017)
- [NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections](https://mlsanity.com/paper/2008.02268/) (2020)
- [NeRF++: Analyzing and Improving Neural Radiance Fields](https://mlsanity.com/paper/2010.07492/) (2020)
- [MVSNet: Depth Inference for Unstructured Multi-view Stereo](https://mlsanity.com/paper/1804.02505/) (2018)
- [Depth-supervised NeRF: Fewer Views and Faster Training for Free](https://mlsanity.com/paper/2107.02791/) (2021)
- [Block-NeRF: Scalable Large Scene Neural View Synthesis](https://mlsanity.com/paper/2202.05263/) (2022)
- [Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fields](https://mlsanity.com/paper/2304.06706/) (2023)
- [BARF: Bundle-Adjusting Neural Radiance Fields](https://mlsanity.com/paper/2104.06405/) (2021)
- [3D Packing for Self-Supervised Monocular Depth Estimation](https://mlsanity.com/paper/1905.02693/) (2019)
- [Mega-NeRF: Scalable Construction of Large-Scale NeRFs for Virtual Fly-Throughs](https://mlsanity.com/paper/2112.10703/) (2021)
- [UniSim: A Neural Closed-Loop Sensor Simulator](https://mlsanity.com/paper/2308.01898/) (2023)
- [Urban Radiance Fields](https://mlsanity.com/paper/2111.14643/) (2021)
- [Self-Calibrating Neural Radiance Fields](https://mlsanity.com/paper/2108.13826/) (2021)
- [SPARF: Neural Radiance Fields from Sparse and Noisy Poses](https://mlsanity.com/paper/2211.11738/) (2022)
- [Panoptic NeRF: 3D-to-2D Label Transfer for Panoptic Urban Scene Segmentation](https://mlsanity.com/paper/2203.15224/) (2022)
- [SUDS: Scalable Urban Dynamic Scenes](https://mlsanity.com/paper/2303.14536/) (2023)
- [Point-Based Multi-View Stereo Network](https://mlsanity.com/paper/1908.04422/) (2019)
- [F2-NeRF: Fast Neural Radiance Field Training with Free Camera Trajectories](https://mlsanity.com/paper/2303.15951/) (2023)
- [S-NeRF: Neural Radiance Fields for Street Views](https://mlsanity.com/paper/2303.00749/) (2023)
- [Neural Fields meet Explicit Geometric Representation for Inverse Rendering of Urban Scenes](https://mlsanity.com/paper/2304.03266/) (2023)
- [Multi-view Depth Estimation using Epipolar Spatio-Temporal Networks](https://mlsanity.com/paper/2011.13118/) (2020)
- [Waymo Open Dataset: Panoramic Video Panoptic Segmentation](https://mlsanity.com/paper/2206.07704/) (2022)
- [Standing Between Past and Future: Spatio-Temporal Modeling for Multi-Camera 3D Multi-Object Tracking](https://mlsanity.com/paper/2302.03802/) (2023)
- [VA-DepthNet: A Variational Approach to Single Image Depth Prediction](https://mlsanity.com/paper/2302.06556/) (2023)

## Built on by (most cited first, 23 of 23)

- [Street Gaussians: Modeling Dynamic Urban Scenes with Gaussian Splatting](https://mlsanity.com/paper/2401.01339/) (2024)
- [GaussianPro: 3D Gaussian Splatting with Progressive Propagation](https://mlsanity.com/paper/2402.14650/) (2024)
- [DriveDreamer4D: World Models Are Effective Data Machines for 4D Driving Scene Representation](https://mlsanity.com/paper/2410.13571/) (2024)
- [ReconDreamer: Crafting World Models for Driving Scene Reconstruction via Online Restoration](https://mlsanity.com/paper/2411.19548/) (2024)
- [ReconDreamer++: Harmonizing Generative and Reconstructive Models for Driving Scene Representation](https://mlsanity.com/paper/2503.18438/) (2025)
- [EVolSplat: Efficient Volume-based Gaussian Splatting for Urban View Synthesis](https://mlsanity.com/paper/2503.20168/) (2025)
- [Neural Radiance Field in Autonomous Driving: A Survey](https://mlsanity.com/paper/2404.13816/) (2024)
- [GGS: Generalizable Gaussian Splatting for Lane Switching in Autonomous Driving](https://mlsanity.com/paper/2409.02382/) (2024)
- [DC-Gaussian: Improving 3D Gaussian Splatting for Reflective Dash Cam Videos](https://mlsanity.com/paper/2405.17705/) (2024)
- [TrackNeRF: Bundle Adjusting NeRF from Sparse and Noisy Views via Feature Tracks](https://mlsanity.com/paper/2408.10739/) (2024)
- [GVKF: Gaussian Voxel Kernel Functions for Highly Efficient Surface Reconstruction in Open Scenes](https://mlsanity.com/paper/2411.01853/) (2024)
- [XLD: A Cross-Lane Dataset for Benchmarking Novel Driving View Synthesis](https://mlsanity.com/paper/2406.18360/) (2024)
- [Graph-Guided Scene Reconstruction from Images with 3D Gaussian Splatting](https://mlsanity.com/paper/2502.17377/) (2025)
- [Efficient Depth-Guided Urban View Synthesis](https://mlsanity.com/paper/2407.12395/) (2024)
- [SDGE: Stereo Guided Depth Estimation for 360^circ Camera Sets](https://mlsanity.com/paper/2402.11791/) (2024)
- [Car-GS: Addressing Reflective and Transparent Surface Challenges in 3D Car Reconstruction](https://mlsanity.com/paper/2501.11020/) (2025)
- [Refining Ground Truth Poses in Autonomous Driving Datasets via Neural Rendering](https://mlsanity.com/paper/2504.15776/) (2025)
- [Gaussian Time Machine: A Real-Time Rendering Methodology for Time-Variant Appearances](https://mlsanity.com/paper/2405.13694/) (2024)
- [S3R-GS: Streamlining the Pipeline for Large-Scale Street Scene Reconstruction](https://mlsanity.com/paper/2503.08217/) (2025)
- [FaithFusion: Harmonizing Reconstruction and Generation via Pixel-wise Information Gain](https://mlsanity.com/paper/2511.21113/) (2025)
- [ScenDi: 3D-to-2D Scene Diffusion Cascades for Urban Generation](https://mlsanity.com/paper/2601.15221/) (2026)
- [Data-driven Camera and Lidar Simulation Models for Autonomous Driving: A Review from Generative Models to Volume Renderers](https://mlsanity.com/paper/2402.10079/) (2024)
- [MaintaAvatar: A Maintainable Avatar Based on Neural Radiance Fields by Continual Learning](https://mlsanity.com/paper/2502.02372/) (2025)

## Abstract (arXiv)

Multi-camera setups find widespread use across various applications, such as autonomous driving, as they greatly expand sensing capabilities. Despite the fast development of Neural radiance field (NeRF) techniques and their wide applications in both indoor and outdoor scenes, applying NeRF to multi-camera systems remains very challenging. This is primarily due to the inherent under-calibration issues in multi-camera setup, including inconsistent imaging effects stemming from separately calibrated image signal processing units in diverse cameras, and system errors arising from mechanical vibrations during driving that affect relative camera poses. In this paper, we present UC-NeRF, a novel method tailored for novel view synthesis in under-calibrated multi-view camera systems. Firstly, we propose a layer-based color correction to rectify the color inconsistency in different image regions. Second, we propose virtual warping to generate more viewpoint-diverse but color-consistent virtual views for color correction and 3D recovery. Finally, a spatiotemporally constrained pose refinement is designed for more robust and accurate pose calibration in multi-camera systems. Our method not only achieves state-of-the-art performance of novel view synthesis in multi-camera setups, but also effectively facilitates depth estimation in large-scale outdoor scenes with the synthesized novel views.