# MOISST: Multimodal Optimization of Implicit Scene for SpatioTemporal calibration

arXiv: https://arxiv.org/abs/2303.03056 · HTML: https://mlsanity.com/paper/2303.03056/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 35 references that are in the map.
- First posted on arXiv 6 Mar 2023.

## Builds on (most cited first, 16 of 16)

- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields](https://mlsanity.com/paper/2111.12077/) (2021)
- [ORB-SLAM: a Versatile and Accurate Monocular SLAM System](https://mlsanity.com/paper/1502.00956/) (2015)
- [BARF: Bundle-Adjusting Neural Radiance Fields](https://mlsanity.com/paper/2104.06405/) (2021)
- [RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs](https://mlsanity.com/paper/2112.00724/) (2021)
- [KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D](https://mlsanity.com/paper/2109.13410/) (2021)
- [iMAP: Implicit Mapping and Positioning in Real-Time](https://mlsanity.com/paper/2103.12352/) (2021)
- [Urban Radiance Fields](https://mlsanity.com/paper/2111.14643/) (2021)
- [Self-Calibrating Neural Radiance Fields](https://mlsanity.com/paper/2108.13826/) (2021)
- [3D-CVF: Generating Joint Camera and LiDAR Features Using Cross-View Spatial Feature Fusion for 3D Object Detection](https://mlsanity.com/paper/2004.12636/) (2020)
- [Dex-NeRF: Using a Neural Radiance Field to Grasp Transparent Objects](https://mlsanity.com/paper/2110.14217/) (2021)
- [LENS: Localization enhanced by NeRF synthesis](https://mlsanity.com/paper/2110.06558/) (2021)
- [RegNet: Multimodal Sensor Registration Using Deep Neural Networks](https://mlsanity.com/paper/1707.03167/) (2017)
- [Pixel-level Extrinsic Self Calibration of High Resolution LiDAR and Camera in Targetless Environments](https://mlsanity.com/paper/2103.01627/) (2021)
- [Spatiotemporal Camera-LiDAR Calibration: A Targetless and Structureless Approach](https://mlsanity.com/paper/2001.06175/) (2020)

## Built on by (most cited first, 19 of 19)

- [DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenes](https://mlsanity.com/paper/2312.07920/) (2023)
- [Deep Learning for Camera Calibration and Beyond: A Survey](https://mlsanity.com/paper/2303.10559/) (2023)
- [Exploring Multi-modal Neural Scene Representations With Applications on Thermal Imaging](https://mlsanity.com/paper/2403.11865/) (2024)
- [RoDUS: Robust Decomposition of Static and Dynamic Elements in Urban Scenes](https://mlsanity.com/paper/2403.09419/) (2024)
- [SOAC: Spatio-Temporal Overlap-Aware Multi-Sensor Calibration using Neural Radiance Fields](https://mlsanity.com/paper/2311.15803/) (2023)
- [3DGS-Calib: 3D Gaussian Splatting for Multimodal SpatioTemporal Calibration](https://mlsanity.com/paper/2403.11577/) (2024)
- [OASim: an Open and Adaptive Simulator based on Neural Rendering for Autonomous Driving](https://mlsanity.com/paper/2402.03830/) (2024)
- [UniCal: Unified Neural Sensor Calibration](https://mlsanity.com/paper/2409.18953/) (2024)
- [ViiNeuS: Volumetric Initialization for Implicit Neural Surface reconstruction of urban scenes with limited image overlap](https://mlsanity.com/paper/2403.10344/) (2024)
- [Refining Ground Truth Poses in Autonomous Driving Datasets via Neural Rendering](https://mlsanity.com/paper/2504.15776/) (2025)
- [DrivingGaussian++: Towards Realistic Reconstruction and Editable Simulation for Surrounding Dynamic Driving Scenes](https://mlsanity.com/paper/2508.20965/) (2025)
- [BEVCALIB: LiDAR-Camera Calibration via Geometry-Guided Bird's-Eye View Representations](https://mlsanity.com/paper/2506.02587/) (2025)
- [UniQueR: Unified Query-based Feedforward 3D Reconstruction](https://mlsanity.com/paper/2603.22851/) (2026)
- [J-NeuS: Joint field optimization for Neural Surface reconstruction in urban scenes with limited image overlap](https://mlsanity.com/paper/2501.03932/) (2025)
- [Robust LiDAR-Camera Calibration with 2D Gaussian Splatting](https://mlsanity.com/paper/2504.00525/) (2025)
- [Targetless LiDAR-Camera Calibration with Neural Gaussian Splatting](https://mlsanity.com/paper/2504.04597/) (2025)
- [NeRF-enabled Analysis-Through-Synthesis for ISAR Imaging of Small Everyday Objects with Sparse and Noisy UWB Radar Data](https://mlsanity.com/paper/2410.10085/) (2024)
- [Native-Domain Cross-Attention for Camera-LiDAR Extrinsic Calibration Under Large Initial Perturbations](https://mlsanity.com/paper/2603.29414/) (2026)
- [SpectralSplat: Appearance-Disentangled Feed-Forward Gaussian Splatting for Driving Scenes](https://mlsanity.com/paper/2604.03462/) (2026)

## Abstract (arXiv)

With the recent advances in autonomous driving and the decreasing cost of LiDARs, the use of multimodal sensor systems is on the rise. However, in order to make use of the information provided by a variety of complimentary sensors, it is necessary to accurately calibrate them. We take advantage of recent advances in computer graphics and implicit volumetric scene representation to tackle the problem of multi-sensor spatial and temporal calibration. Thanks to a new formulation of the Neural Radiance Field (NeRF) optimization, we are able to jointly optimize calibration parameters along with scene representation based on radiometric and geometric measurements. Our method enables accurate and robust calibration from data captured in uncontrolled and unstructured urban environments, making our solution more scalable than existing calibration solutions. We demonstrate the accuracy and robustness of our method in urban scenes typically encountered in autonomous driving scenarios.