# SpectralZoom: Efficient Segmentation with an Adaptive Hyperspectral Camera

arXiv: https://arxiv.org/abs/2406.04287 · HTML: https://mlsanity.com/paper/2406.04287/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 39 references that are in the map.
- First posted on arXiv 6 Jun 2024.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Segmenter: Transformer for Semantic Segmentation](https://mlsanity.com/paper/2105.05633/) (2021)
- [SpectralFormer: Rethinking Hyperspectral Image Classification with Transformers](https://mlsanity.com/paper/2107.02988/) (2021)
- [Patch Slimming for Efficient Vision Transformers](https://mlsanity.com/paper/2106.02852/) (2021)
- [Self-supervised Monocular Trained Depth Estimation using Self-attention and Discrete Disparity Volume](https://mlsanity.com/paper/2003.13951/) (2020)
- [Zoom To Learn, Learn To Zoom](https://mlsanity.com/paper/1905.05169/) (2019)
- [Deep Optics for Monocular Depth Estimation and 3D Object Detection](https://mlsanity.com/paper/1904.08601/) (2019)
- [Learning When and Where to Zoom with Deep Reinforcement Learning](https://mlsanity.com/paper/2003.00425/) (2020)
- [Learning Sensor Multiplexing Design through Back-propagation](https://mlsanity.com/paper/1605.07078/) (2016)
- [SpecTr: Spectral Transformer for Hyperspectral Pathology Image Segmentation](https://mlsanity.com/paper/2103.03604/) (2021)
- [Polka Lines: Learning Structured Illumination and Reconstruction for Active Stereo](https://mlsanity.com/paper/2011.13117/) (2020)
- [Measuring the Ripeness of Fruit with Hyperspectral Imaging and Deep Learning](https://mlsanity.com/paper/2104.09808/) (2021)
- [A Fast 3D CNN for Hyperspectral Image Classification](https://mlsanity.com/paper/2004.14152/) (2020)
- [Differentiable Programming for Hyperspectral Unmixing using a Physics-based Dispersion Model](https://mlsanity.com/paper/2007.05996/) (2020)
- [Peanut Maturity Classification using Hyperspectral Imagery](https://mlsanity.com/paper/1910.11122/) (2019)
- [SaccadeCam: Adaptive Visual Attention for Monocular Depth Sensing](https://mlsanity.com/paper/2103.12981/) (2021)

## Built on by (most cited first, 2 of 2)

- [Hyperspectral Sensors and Autonomous Driving: Technologies, Limitations, and Opportunities](https://mlsanity.com/paper/2508.19905/) (2025)
- [Multi-Scale Spectral Attention Module-based Hyperspectral Segmentation in Autonomous Driving Scenarios](https://mlsanity.com/paper/2506.18682/) (2025)

## Abstract (arXiv)

Hyperspectral image segmentation is crucial for many fields such as agriculture, remote sensing, biomedical imaging, battlefield sensing and astronomy. However, the challenge of hyper and multi spectral imaging is its large data footprint. We propose both a novel camera design and a vision transformer-based (ViT) algorithm that alleviate both the captured data footprint and the computational load for hyperspectral segmentation. Our camera is able to adaptively sample image regions or patches at different resolutions, instead of capturing the entire hyperspectral cube at one high resolution. Our segmentation algorithm works in concert with the camera, applying ViT-based segmentation only to adaptively selected patches. We show results both in simulation and on a real hardware platform demonstrating both accurate segmentation results and reduced computational burden.