# UnMix-NeRF: Spectral Unmixing Meets Neural Radiance Fields

arXiv: https://arxiv.org/abs/2506.21884 · HTML: https://mlsanity.com/paper/2506.21884/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 53 references that are in the map.
- First posted on arXiv 27 Jun 2025.

## Builds on (most cited first, 30 of 31)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Nerfstudio: A Modular Framework for Neural Radiance Field Development](https://mlsanity.com/paper/2302.04264/) (2023)
- [Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields](https://mlsanity.com/paper/2112.03907/) (2021)
- [LERF: Language Embedded Radiance Fields](https://mlsanity.com/paper/2303.09553/) (2023)
- [Baking Neural Radiance Fields for Real-Time View Synthesis](https://mlsanity.com/paper/2103.14645/) (2021)
- [NerfAcc: Efficient Sampling Accelerates NeRFs](https://mlsanity.com/paper/2305.04966/) (2023)
- [SeaThru-NeRF: Neural Radiance Fields in Scattering Media](https://mlsanity.com/paper/2304.07743/) (2023)
- [OpenNeRF: Open Set 3D Neural Scene Segmentation with Pixel-Wise Features and Rendered Novel Views](https://mlsanity.com/paper/2404.03650/) (2024)
- [An Augmented Linear Mixing Model to Address Spectral Variability for Hyperspectral Unmixing](https://mlsanity.com/paper/1810.12000/) (2018)
- [LiDAR-NeRF: Novel LiDAR View Synthesis via Neural Radiance Fields](https://mlsanity.com/paper/2304.10406/) (2023)
- [Spectral Variability in Hyperspectral Data Unmixing: A Comprehensive Review](https://mlsanity.com/paper/2001.07307/) (2020)
- [NeRF-RPN: A general framework for object detection in NeRFs](https://mlsanity.com/paper/2211.11646/) (2022)
- [MatFuse: Controllable Material Generation with Diffusion Models](https://mlsanity.com/paper/2308.11408/) (2023)
- [Materialistic: Selecting Similar Materials in Images](https://mlsanity.com/paper/2305.13291/) (2023)
- [Floaters No More: Radiance Field Gradient Scaling for Improved Near-Camera Training](https://mlsanity.com/paper/2305.02756/) (2023)
- [Deep Hyperspectral Unmixing using Transformer Network](https://mlsanity.com/paper/2203.17076/) (2022)
- [Unsupervised Multi-View Object Segmentation Using Radiance Field Propagation](https://mlsanity.com/paper/2210.00489/) (2022)
- [Thermal-NeRF: Neural Radiance Fields from an Infrared Camera](https://mlsanity.com/paper/2403.10340/) (2024)
- [SANeRF-HQ: Segment Anything for NeRF in High Quality](https://mlsanity.com/paper/2312.01531/) (2023)
- [Spec-NeRF: Multi-spectral Neural Radiance Fields](https://mlsanity.com/paper/2310.12987/) (2023)
- [SpectralNeRF: Physically Based Spectral Rendering with Neural Radiance Field](https://mlsanity.com/paper/2312.08692/) (2023)
- [Hyperspectral Neural Radiance Fields](https://mlsanity.com/paper/2403.14839/) (2024)
- [HyperGS: Hyperspectral 3D Gaussian Splatting](https://mlsanity.com/paper/2412.12849/) (2024)
- [Neural Spectro-polarimetric Fields](https://mlsanity.com/paper/2306.12562/) (2023)
- [SAMa: Material-aware 3D Selection and Segmentation](https://mlsanity.com/paper/2411.19322/) (2024)
- [MateRobot: Material Recognition in Wearable Robotics for People with Visual Impairments](https://mlsanity.com/paper/2302.14595/) (2023)
- [ONeRF: Unsupervised 3D Object Segmentation from Multiple Views](https://mlsanity.com/paper/2211.12038/) (2022)
- [MaterialSeg3D: Segmenting Dense Materials from 2D Priors for 3D Assets](https://mlsanity.com/paper/2404.13923/) (2024)

## Built on by (most cited first, 3 of 3)

- [PoInit-of-View: Poisoning Initialization of Views Transfers Across Multiple 3D Reconstruction Systems](https://mlsanity.com/paper/2604.16540/) (2026)
- [Learning Spectral and Polarimetric Clues for One-to-Multimodal Novel View Synthesis](https://mlsanity.com/paper/2607.02372/) (2026)
- [Spectral World Renderer: Unified 3D Hyperspectral Unmixing](https://mlsanity.com/paper/2610.19520/) (2026)

## Abstract (arXiv)

Neural Radiance Field (NeRF)-based segmentation methods focus on object semantics and rely solely on RGB data, lacking intrinsic material properties. This limitation restricts accurate material perception, which is crucial for robotics, augmented reality, simulation, and other applications. We introduce UnMix-NeRF, a framework that integrates spectral unmixing into NeRF, enabling joint hyperspectral novel view synthesis and unsupervised material segmentation. Our method models spectral reflectance via diffuse and specular components, where a learned dictionary of global endmembers represents pure material signatures, and per-point abundances capture their distribution. For material segmentation, we use spectral signature predictions along learned endmembers, allowing unsupervised material clustering. Additionally, UnMix-NeRF enables scene editing by modifying learned endmember dictionaries for flexible material-based appearance manipulation. Extensive experiments validate our approach, demonstrating superior spectral reconstruction and material segmentation to existing methods. Project page: https://www.factral.co/UnMix-NeRF.