# PhotoReg: Photometrically Registering 3D Gaussian Splatting Models

arXiv: https://arxiv.org/abs/2410.05044 · HTML: https://mlsanity.com/paper/2410.05044/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 47 references that are in the map.
- First posted on arXiv 7 Oct 2024.

## Builds on (most cited first, 25 of 25)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [DUSt3R: Geometric 3D Vision Made Easy](https://mlsanity.com/paper/2312.14132/) (2023)
- [Large Scale Adversarial Representation Learning](https://mlsanity.com/paper/1907.02544/) (2019)
- [AnyLoc: Towards Universal Visual Place Recognition](https://mlsanity.com/paper/2308.00688/) (2023)
- [RaDe-GS: Rasterizing Depth in Gaussian Splatting](https://mlsanity.com/paper/2406.01467/) (2024)
- [LoopSplat: Loop Closure by Registering 3D Gaussian Splats](https://mlsanity.com/paper/2408.10154/) (2024)
- [Neural Volume Rendering: NeRF And Beyond](https://mlsanity.com/paper/2101.05204/) (2020)
- [nerf2nerf: Pairwise Registration of Neural Radiance Fields](https://mlsanity.com/paper/2211.01600/) (2022)
- [NeBula: Quest for Robotic Autonomy in Challenging Environments; TEAM CoSTAR at the DARPA Subterranean Challenge](https://mlsanity.com/paper/2103.11470/) (2021)
- [DarkGS: Learning Neural Illumination and 3D Gaussians Relighting for Robotic Exploration in the Dark](https://mlsanity.com/paper/2403.10814/) (2024)
- [DReg-NeRF: Deep Registration for Neural Radiance Fields](https://mlsanity.com/paper/2308.09386/) (2023)
- [Beyond NeRF Underwater: Learning Neural Reflectance Fields for True Color Correction of Marine Imagery](https://mlsanity.com/paper/2304.03384/) (2023)
- [Parallelised Diffeomorphic Sampling-based Motion Planning](https://mlsanity.com/paper/2108.11775/) (2021)
- [RecGS: Removing Water Caustic with Recurrent Gaussian Splatting](https://mlsanity.com/paper/2407.10318/) (2024)
- [Self-supervised Point Set Local Descriptors for Point Cloud Registration](https://mlsanity.com/paper/2003.05199/) (2020)
- [Directional grid maps: modeling multimodal angular uncertainty in dynamic environments](https://mlsanity.com/paper/1809.00498/) (2018)
- [Kernel Trajectory Maps for Multi-Modal Probabilistic Motion Prediction](https://mlsanity.com/paper/1907.05127/) (2019)
- [3D Foundation Models Enable Simultaneous Geometry and Pose Estimation of Grasped Objects](https://mlsanity.com/paper/2407.10331/) (2024)

## Built on by (most cited first, 5 of 5)

- [SIREN: Semantic, Initialization-Free Registration of Multi-Robot Gaussian Splatting Maps](https://mlsanity.com/paper/2502.06519/) (2025)
- [Automated 3D-GS Registration and Fusion via Skeleton Alignment and Gaussian-Adaptive Features](https://mlsanity.com/paper/2507.20480/) (2025)
- [Infinite Leagues Under the Sea: Photorealistic 3D Underwater Terrain Generation by Latent Fractal Diffusion Models](https://mlsanity.com/paper/2503.06784/) (2025)
- [Graph-GSReg: Leveraging 3D Scene Graphs for Gaussian Splatting Registration](https://mlsanity.com/paper/2606.29782/) (2026)
- [CoRef-GS: Cooperative Referring Gaussian Splatting for Multi-Agent Scene Understanding](https://mlsanity.com/paper/2609.20586/) (2026)

## Abstract (arXiv)

Building accurate representations of the environment is critical for intelligent robots to make decisions during deployment. Advances in photorealistic environment models have enabled robots to develop hyper-realistic reconstructions, which can be used to generate images that are intuitive for human inspection. In particular, the recently introduced ac3DGS, which describes the scene with up to millions of primitive ellipsoids, can be rendered in real time. ac3DGS has rapidly gained prominence. However, a critical unsolved problem persists: how can we fuse multiple ac3DGS into a single coherent model? Solving this problem will enable robot teams to jointly build ac3DGS models of their surroundings. A key insight of this work is to leverage the duality between photorealistic reconstructions, which render realistic 2D images from 3D structure, and 3D foundation models, which predict 3D structure from image pairs. To this end, we develop PhotoReg, a framework to register multiple photorealistic ac3DGS models with 3D foundation models. As ac3DGS models are generally built from monocular camera images, they have arbitrary scale. To resolve this, PhotoReg actively enforces scale consistency among the different ac3DGS models by considering depth estimates within these models. Then, the alignment is iteratively refined with fine-grained photometric losses to produce high-quality fused ac3DGS models. We rigorously evaluate PhotoReg on both standard benchmark datasets and our custom-collected datasets, including with two quadruped robots. The code is released at urlziweny11.github.io/photoreg.