# SpecTr: Spectral Transformer for Hyperspectral Pathology Image Segmentation

arXiv: https://arxiv.org/abs/2103.03604 · HTML: https://mlsanity.com/paper/2103.03604/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 19 references that are in the map.
- First posted on arXiv 5 Mar 2021.

## Builds on (most cited first, 12 of 12)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation](https://mlsanity.com/paper/2102.04306/) (2021)
- [UNet++: A Nested U-Net Architecture for Medical Image Segmentation](https://mlsanity.com/paper/1807.10165/) (2018)
- [3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation](https://mlsanity.com/paper/1606.06650/) (2016)
- [Attention U-Net: Learning Where to Look for the Pancreas](https://mlsanity.com/paper/1804.03999/) (2018)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Image Transformer](https://mlsanity.com/paper/1802.05751/) (2018)
- [Medical Transformer: Gated Axial-Attention for Medical Image Segmentation](https://mlsanity.com/paper/2102.10662/) (2021)
- [Adaptively Sparse Transformers](https://mlsanity.com/paper/1909.00015/) (2019)

## Built on by (most cited first, 23 of 23)

- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [Transformers in Medical Imaging: A Survey](https://mlsanity.com/paper/2201.09873/) (2022)
- [nnFormer: Interleaved Transformer for Volumetric Segmentation](https://mlsanity.com/paper/2109.03201/) (2021)
- [TransFG: A Transformer Architecture for Fine-grained Recognition](https://mlsanity.com/paper/2103.07976/) (2021)
- [Transformers in Medical Image Analysis: A Review](https://mlsanity.com/paper/2202.12165/) (2022)
- [Transforming medical imaging with Transformers? A comparative review of key properties, current progresses, and future perspectives](https://mlsanity.com/paper/2206.01136/) (2022)
- [3D Vision with Transformers: A Survey](https://mlsanity.com/paper/2208.04309/) (2022)
- [Transformers in Healthcare: A Survey](https://mlsanity.com/paper/2307.00067/) (2023)
- [A survey on attention mechanisms for medical applications: are we moving towards better algorithms?](https://mlsanity.com/paper/2204.12406/) (2022)
- [EdgeLoRA: An Efficient Multi-Tenant LLM Serving System on Edge Devices](https://mlsanity.com/paper/2507.01438/) (2025)
- [The Lighter The Better: Rethinking Transformers in Medical Image Segmentation Through Adaptive Pruning](https://mlsanity.com/paper/2206.14413/) (2022)
- [Attention Mechanisms in Medical Image Segmentation: A Survey](https://mlsanity.com/paper/2305.17937/) (2023)
- [Masked LoGoNet: Fast and Accurate 3D Image Analysis for Medical Domain](https://mlsanity.com/paper/2402.06190/) (2024)
- [TrustEMG-Net: Using Representation-Masking Transformer with U-Net for Surface Electromyography Enhancement](https://mlsanity.com/paper/2410.03843/) (2024)
- [Clover-2: Accurate Inference for Regressive Lightweight Speculative Decoding](https://mlsanity.com/paper/2408.00264/) (2024)
- [HResFormer: Hybrid Residual Transformer for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/2412.11458/) (2024)
- [MultiNet with Transformers: A Model for Cancer Diagnosis Using Images](https://mlsanity.com/paper/2301.09007/) (2023)
- [SpectralZoom: Efficient Segmentation with an Adaptive Hyperspectral Camera](https://mlsanity.com/paper/2406.04287/) (2024)
- [Robust Tumor Segmentation with Hyperspectral Imaging and Graph Neural Networks](https://mlsanity.com/paper/2311.11782/) (2023)
- [MDN: Mamba-Driven Dualstream Network For Medical Hyperspectral Image Segmentation](https://mlsanity.com/paper/2502.17255/) (2025)
- [Revisiting the dynamics of Bose-Einstein condensates in a double well by deep learning with a hybrid network](https://mlsanity.com/paper/2104.14657/) (2021)
- [MCPA: Multi-scale Cross Perceptron Attention Network for 2D Medical Image Segmentation](https://mlsanity.com/paper/2307.14588/) (2023)
- [A Hyperspectral Imaging Dataset and Methodology for Intraoperative Pixel-Wise Classification of Metastatic Colon Cancer in the Liver](https://mlsanity.com/paper/2411.06969/) (2024)

## Abstract (arXiv)

Hyperspectral imaging (HSI) unlocks the huge potential to a wide variety of applications relied on high-precision pathology image segmentation, such as computational pathology and precision medicine. Since hyperspectral pathology images benefit from the rich and detailed spectral information even beyond the visible spectrum, the key to achieve high-precision hyperspectral pathology image segmentation is to felicitously model the context along high-dimensional spectral bands. Inspired by the strong context modeling ability of transformers, we hereby, for the first time, formulate the contextual feature learning across spectral bands for hyperspectral pathology image segmentation as a sequence-to-sequence prediction procedure by transformers. To assist spectral context learning procedure, we introduce two important strategies: (1) a sparsity scheme enforces the learned contextual relationship to be sparse, so as to eliminates the distraction from the redundant bands; (2) a spectral normalization, a separate group normalization for each spectral band, mitigates the nuisance caused by heterogeneous underlying distributions of bands. We name our method Spectral Transformer (SpecTr), which enjoys two benefits: (1) it has a strong ability to model long-range dependency among spectral bands, and (2) it jointly explores the spatial-spectral features of HSI. Experiments show that SpecTr outperforms other competing methods in a hyperspectral pathology image segmentation benchmark without the need of pre-training. Code is available at https://github.com/hfut-xc-yun/SpecTr.