# Hyperspectral Adapter for Semantic Segmentation with Vision Foundation Models

arXiv: https://arxiv.org/abs/2509.20107 · HTML: https://mlsanity.com/paper/2509.20107/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 35 references that are in the map.
- First posted on arXiv 24 Sep 2025.

## Builds on (most cited first, 24 of 24)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Unified Perceptual Parsing for Scene Understanding](https://mlsanity.com/paper/1807.10221/) (2018)
- [Vision Transformer Adapter for Dense Predictions](https://mlsanity.com/paper/2205.08534/) (2022)
- [SpectralFormer: Rethinking Hyperspectral Image Classification with Transformers](https://mlsanity.com/paper/2107.02988/) (2021)
- [Deep Learning for Hyperspectral Image Classification: An Overview](https://mlsanity.com/paper/1910.12861/) (2019)
- [StitchFusion: Weaving Any Visual Modalities to Enhance Multimodal Semantic Segmentation](https://mlsanity.com/paper/2408.01343/) (2024)
- [BEVCar: Camera-Radar Fusion for BEV Map and Object Segmentation](https://mlsanity.com/paper/2403.11761/) (2024)
- [Few-Shot Panoptic Segmentation With Foundation Models](https://mlsanity.com/paper/2309.10726/) (2023)
- [UniRGB-IR: A Unified Framework for Visible-Infrared Semantic Tasks via Adapter Tuning](https://mlsanity.com/paper/2404.17360/) (2024)
- [AmodalSynthDrive: A Synthetic Amodal Perception Dataset for Autonomous Driving](https://mlsanity.com/paper/2309.06547/) (2023)
- [Panoptic Out-of-Distribution Segmentation](https://mlsanity.com/paper/2310.11797/) (2023)
- [Neural Architecture Search for Dense Prediction Tasks in Computer Vision](https://mlsanity.com/paper/2202.07242/) (2022)
- [Deep Learning Techniques for Hyperspectral Image Analysis in Agriculture: A Review](https://mlsanity.com/paper/2304.13880/) (2023)
- [HS3-Bench: A Benchmark and Strong Baseline for Hyperspectral Semantic Segmentation in Driving Scenarios](https://mlsanity.com/paper/2409.11205/) (2024)
- [HSI-Drive v2.0: More Data for New Challenges in Scene Understanding for Autonomous Driving](https://mlsanity.com/paper/2411.17530/) (2024)
- [Syn-Mediverse: A Multimodal Synthetic Dataset for Intelligent Scene Understanding of Healthcare Facilities](https://mlsanity.com/paper/2308.03193/) (2023)
- [Progressive Multi-Modal Fusion for Robust 3D Object Detection](https://mlsanity.com/paper/2410.07475/) (2024)
- [Label-Efficient LiDAR Semantic Segmentation with 2D-3D Vision Transformer Adapters](https://mlsanity.com/paper/2503.03299/) (2025)
- [Amodal Optical Flow](https://mlsanity.com/paper/2311.07761/) (2023)
- [A Good Foundation is Worth Many Labels: Label-Efficient Panoptic Segmentation](https://mlsanity.com/paper/2405.19035/) (2024)
- [Hyperspectral City V1.0 Dataset and Benchmark](https://mlsanity.com/paper/1907.10270/) (2019)

## Built on by (most cited first, 7 of 7)

- [Bridging Perspectives: Foundation Model Guided BEV Maps for 3D Object Detection and Tracking](https://mlsanity.com/paper/2510.10287/) (2025)
- [HyperVision: A Channel-Adaptive Ground-Based Hyperspectral Vision Pre-trained Backbone](https://mlsanity.com/paper/2605.17286/) (2026)
- [Cross-Domain Transfer of Hyperspectral Foundation Models](https://mlsanity.com/paper/2604.26478/) (2026)
- [Hyp2Former: Hierarchy-Aware Hyperbolic Embeddings for Open-Set Panoptic Segmentation](https://mlsanity.com/paper/2605.02580/) (2026)
- [Leveraging Vision-Based Point Cloud Map Priors for Camera-Based 3D Object Detection and Online Vectorized HD Mapping](https://mlsanity.com/paper/2609.26325/) (2026)
- [Perception Beyond the Visible: Incompleteness-Aware Scene Understanding for Robot Autonomy](https://mlsanity.com/paper/2610.18670/) (2026)
- [SPECTRA-VLA: Base-Fine Hyperspectral Token Transfer at Frozen Vision-Language-Action Interfaces](https://mlsanity.com/paper/2610.19651/) (2026)

## Abstract (arXiv)

Hyperspectral imaging (HSI) captures spatial information along with dense spectral measurements across numerous narrow wavelength bands. This rich spectral content has the potential to facilitate robust robotic perception, particularly in environments with complex material compositions, varying illumination, or other visually challenging conditions. However, current HSI semantic segmentation methods underperform due to their reliance on architectures and learning frameworks optimized for RGB inputs. In this work, we propose a novel hyperspectral adapter that leverages pretrained vision foundation models to effectively learn from hyperspectral data. Our architecture incorporates a spectral transformer and a spectrum-aware spatial prior module to extract rich spatial-spectral features. Additionally, we introduce a modality-aware interaction block that facilitates effective integration of hyperspectral representations and frozen vision Transformer features through dedicated extraction and injection mechanisms. Extensive evaluations on three benchmark autonomous driving datasets demonstrate that our architecture achieves state-of-the-art semantic segmentation performance while directly using HSI inputs, outperforming both vision-based and hyperspectral segmentation methods. We make the code available at https://hsi-adapter.cs.uni-freiburg.de.