# Unpaired RGB-Thermal Gaussian-Splatting Using Visual Geometric Transformers

arXiv: https://arxiv.org/abs/2606.05491 · HTML: https://mlsanity.com/paper/2606.05491/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 22 references that are in the map.
- First posted on arXiv 3 Jun 2026.

## Builds on (most cited first, 16 of 16)

- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [DUSt3R: Geometric 3D Vision Made Easy](https://mlsanity.com/paper/2312.14132/) (2023)
- [VGGT: Visual Geometry Grounded Transformer](https://mlsanity.com/paper/2503.11651/) (2025)
- [Nerfstudio: A Modular Framework for Neural Radiance Field Development](https://mlsanity.com/paper/2302.04264/) (2023)
- [MASt3R-SfM: a Fully-Integrated Solution for Unconstrained Structure-from-Motion](https://mlsanity.com/paper/2409.19152/) (2024)
- [MAGSAC: marginalizing sample consensus](https://mlsanity.com/paper/1803.07469/) (2018)
- [XoFTR: Cross-modal Feature Matching Transformer](https://mlsanity.com/paper/2404.09692/) (2024)
- [Thermal-NeRF: Neural Radiance Fields from an Infrared Camera](https://mlsanity.com/paper/2403.10340/) (2024)
- [Exploring Multi-modal Neural Scene Representations With Applications on Thermal Imaging](https://mlsanity.com/paper/2403.11865/) (2024)
- [High-Fidelity SLAM Using Gaussian Splatting with Rendering-Guided Densification and Regularized Optimization](https://mlsanity.com/paper/2403.12535/) (2024)
- [ThermalGaussian: Thermal 3D Gaussian Splatting](https://mlsanity.com/paper/2409.07200/) (2024)
- [Why are thermal images blurry](https://mlsanity.com/paper/2307.15800/) (2023)
- [HyperGS: Hyperspectral 3D Gaussian Splatting](https://mlsanity.com/paper/2412.12849/) (2024)
- [3QFP: Efficient neural implicit surface reconstruction using Tri-Quadtrees and Fourier feature Positional encoding](https://mlsanity.com/paper/2401.07164/) (2024)
- [Exploiting Semantic Scene Reconstruction for Estimating Building Envelope Characteristics](https://mlsanity.com/paper/2410.22383/) (2024)
- [Thermoxels: a voxel-based method to generate simulation-ready 3D thermal models](https://mlsanity.com/paper/2504.04448/) (2025)

## Abstract (arXiv)

Multi-modal novel view synthesis (NVS) combining RGB and thermal imagery enables precise 3D scene reconstruction with visual and thermal information. However, existing methods typically rely on precisely calibrated RGB-thermal image pairs or stereo setups, limiting scalability and practical deployment. To address this, we introduce a framework for unpaired RGB-thermal NVS that leverages VGGT, a 3D feed-forward transformer architecture, to independently estimate camera poses for each modality. The pose sets are then aligned using the Procrustes algorithm with a cross-modal feature matcher, enabling joint registration without paired calibration. Building on this alignment, we further propose a multi-modal 3D Gaussian Splatting approach that learns directly from unpaired RGB and thermal images. Experiments on diverse scenes demonstrate that our method achieves competitive performance in thermal view synthesis while maintaining RGB fidelity. Moreover, we show that existing reconstruction approaches can produce modality-specific reconstructions that lack cross-modal consistency. We thus introduce a benchmarking framework to rigorously evaluate both per-modality image synthesis and the multi-modal coherence of reconstructed scenes.