# Multi-Spectral Gaussian Splatting with Neural Color Representation

arXiv: https://arxiv.org/abs/2506.03407 · HTML: https://mlsanity.com/paper/2506.03407/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 66 references that are in the map.
- First posted on arXiv 3 Jun 2025.

## Builds on (most cited first, 26 of 26)

- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [NeRF++: Analyzing and Improving Neural Radiance Fields](https://mlsanity.com/paper/2010.07492/) (2020)
- [Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis](https://mlsanity.com/paper/2308.09713/) (2023)
- [Neural Point-Based Graphics](https://mlsanity.com/paper/1906.08240/) (2019)
- [State of the Art on Neural Rendering](https://mlsanity.com/paper/2004.03805/) (2020)
- [A Hierarchical 3D Gaussian Representation for Real-Time Rendering of Very Large Datasets](https://mlsanity.com/paper/2406.12080/) (2024)
- [VastGaussian: Vast 3D Gaussians for Large Scene Reconstruction](https://mlsanity.com/paper/2402.17427/) (2024)
- [Point-Based Neural Rendering with Per-View Optimization](https://mlsanity.com/paper/2109.02369/) (2021)
- [Neural Point Catacaustics for Novel-View Synthesis of Reflections](https://mlsanity.com/paper/2301.01087/) (2023)
- [Gaussian Splatting: 3D Reconstruction and Novel View Synthesis, a Review](https://mlsanity.com/paper/2405.03417/) (2024)
- [Efficient Perspective-Correct 3D Gaussian Splatting Using Hybrid Transparency](https://mlsanity.com/paper/2410.08129/) (2024)
- [TRIPS: Trilinear Point Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2401.06003/) (2024)
- [VR-Splatting: Foveated Radiance Field Rendering via 3D Gaussian Splatting and Neural Points](https://mlsanity.com/paper/2410.17932/) (2024)
- [FruitNeRF: A Unified Neural Radiance Field based Fruit Counting Framework](https://mlsanity.com/paper/2408.06190/) (2024)
- [PAg-NeRF: Towards fast and efficient end-to-end panoptic 3D representations for agricultural robotics](https://mlsanity.com/paper/2309.05339/) (2023)
- [Exploring Multi-modal Neural Scene Representations With Applications on Thermal Imaging](https://mlsanity.com/paper/2403.11865/) (2024)
- [Multi-sensor large-scale dataset for multi-view 3D reconstruction](https://mlsanity.com/paper/2203.06111/) (2022)
- [VET: Visual Error Tomography for Point Cloud Completion and High-Quality Neural Rendering](https://mlsanity.com/paper/2311.04634/) (2023)
- [SpectralNeRF: Physically Based Spectral Rendering with Neural Radiance Field](https://mlsanity.com/paper/2312.08692/) (2023)
- [Feature Splatting for Better Novel View Synthesis with Low Overlap](https://mlsanity.com/paper/2405.15518/) (2024)
- [AgriNeRF: Neural Radiance Fields for Agriculture in Challenging Lighting Conditions](https://mlsanity.com/paper/2409.15487/) (2024)
- [LiveNVS: Neural View Synthesis on Live RGB-D Streams](https://mlsanity.com/paper/2311.16668/) (2023)
- [PanicleNeRF: low-cost, high-precision in-field phenotypingof rice panicles with smartphone](https://mlsanity.com/paper/2408.02053/) (2024)

## Built on by (most cited first, 5 of 5)

- [Towards Integrating Multi-Spectral Imaging with Gaussian Splatting](https://mlsanity.com/paper/2509.00989/) (2025)
- [OmniPlantSeg: Species Agnostic 3D Point Cloud Organ Segmentation for High-Resolution Plant Phenotyping Across Modalities](https://mlsanity.com/paper/2509.21038/) (2025)
- [Fusing in 3D: Free-Viewpoint Fusion Rendering with a 3D Infrared-Visible Scene Representation](https://mlsanity.com/paper/2601.12697/) (2026)
- [ThermoSplat: Cross-Modal 3D Gaussian Splatting with Feature Modulation and Geometry Decoupling](https://mlsanity.com/paper/2601.15897/) (2026)
- [Spectral World Renderer: Unified 3D Hyperspectral Unmixing](https://mlsanity.com/paper/2610.19520/) (2026)

## Abstract (arXiv)

We present MS-Splatting -- a multi-spectral 3D Gaussian Splatting (3DGS) framework that is able to generate multi-view consistent novel views from images of multiple, independent cameras with different spectral domains. In contrast to previous approaches, our method does not require cross-modal camera calibration and is versatile enough to model a variety of different spectra, including thermal and near-infra red, without any algorithmic changes. Unlike existing 3DGS-based frameworks that treat each modality separately (by optimizing per-channel spherical harmonics) and therefore fail to exploit the underlying spectral and spatial correlations, our method leverages a novel neural color representation that encodes multi-spectral information into a learned, compact, per-splat feature embedding. A shallow multi-layer perceptron (MLP) then decodes this embedding to obtain spectral color values, enabling joint learning of all bands within a unified representation. Our experiments show that this simple yet effective strategy is able to improve multi-spectral rendering quality, while also leading to improved per-spectra rendering quality over state-of-the-art methods. We demonstrate the effectiveness of this new technique in agricultural applications to render vegetation indices, such as normalized difference vegetation index (NDVI).