# SegMAN: Omni-scale Context Modeling with State Space Models and Local Attention for Semantic Segmentation

arXiv: https://arxiv.org/abs/2412.11890 · HTML: https://mlsanity.com/paper/2412.11890/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 58 of its 69 references that are in the map.
- First posted on arXiv 16 Dec 2024.

## Builds on (most cited first, 30 of 58)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Pyramid Scene Parsing Network](https://mlsanity.com/paper/1612.01105/) (2016)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Unified Perceptual Parsing for Scene Understanding](https://mlsanity.com/paper/1807.10221/) (2018)
- [Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network](https://mlsanity.com/paper/1609.05158/) (2016)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model](https://mlsanity.com/paper/2401.09417/) (2024)
- [Dual Attention Network for Scene Segmentation](https://mlsanity.com/paper/1809.02983/) (2018)
- [VMamba: Visual State Space Model](https://mlsanity.com/paper/2401.10166/) (2024)
- [Designing Network Design Spaces](https://mlsanity.com/paper/2003.13678/) (2020)
- [COCO-Stuff: Thing and Stuff Classes in Context](https://mlsanity.com/paper/1612.03716/) (2016)
- [Simplified State Space Layers for Sequence Modeling](https://mlsanity.com/paper/2208.04933/) (2022)
- [PVT v2: Improved Baselines with Pyramid Vision Transformer](https://mlsanity.com/paper/2106.13797/) (2021)
- [Image Segmentation Using Deep Learning: A Survey](https://mlsanity.com/paper/2001.05566/) (2020)
- [Panoptic Feature Pyramid Networks](https://mlsanity.com/paper/1901.02446/) (2019)
- [Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layers](https://mlsanity.com/paper/2110.13985/) (2021)

## Built on by (most cited first, 30 of 31)

- [Image Generators are Generalist Vision Learners](https://mlsanity.com/paper/2604.20329/) (2026)
- [OverLoCK: An Overview-first-Look-Closely-next ConvNet with Context-Mixing Dynamic Kernels](https://mlsanity.com/paper/2502.20087/) (2025)
- [Vision Transformers Need More Than Registers](https://mlsanity.com/paper/2602.22394/) (2026)
- [PANORAMA: The Rise of Omnidirectional Vision in the Embodied AI Era](https://mlsanity.com/paper/2509.12989/) (2025)
- [Distilled Large Language Model-Driven Dynamic Sparse Expert Activation Mechanism](https://mlsanity.com/paper/2603.26735/) (2026)
- [Diffusion Model as a Generalist Segmentation Learner](https://mlsanity.com/paper/2604.24575/) (2026)
- [SLARM: Streaming and Language-Aligned Reconstruction Model for Dynamic Scenes](https://mlsanity.com/paper/2603.22893/) (2026)
- [HydraMamba: Multi-Head State Space Model for Global Point Cloud Learning](https://mlsanity.com/paper/2507.19778/) (2025)
- [ARTA: Adaptive Mixed-Resolution Token Allocation for Efficient Dense Feature Extraction](https://mlsanity.com/paper/2603.26258/) (2026)
- [A2Mamba: Attention-augmented State Space Models for Visual Recognition](https://mlsanity.com/paper/2507.16624/) (2025)
- [Segment Any Vehicle: Semantic and Visual Context Driven SAM and A Benchmark](https://mlsanity.com/paper/2508.04260/) (2025)
- [Parameters as Experts: Adapting Vision Models with Dynamic Parameter Routing](https://mlsanity.com/paper/2602.06862/) (2026)
- [LenghuSky-8: An 8-Year All-Sky Cloud Dataset with Star-Aware Masks and Alt-Az Calibration for Segmentation and Nowcasting](https://mlsanity.com/paper/2603.16429/) (2026)
- [XSeg: A Large-scale X-ray Contraband Segmentation Benchmark For Real-World Security Screening](https://mlsanity.com/paper/2604.03706/) (2026)
- [RSGMamba: Reliability-Aware Self-Gated State Space Model for Multimodal Semantic Segmentation](https://mlsanity.com/paper/2604.12319/) (2026)
- [Training-free Cross-domain Few-shot Segmentation via Robust Semantic Representation and Matching](https://mlsanity.com/paper/2606.24297/) (2026)
- [Vision Non-Causal Trapezoidal Mamba: Eliminating Directional Scanning in Vision SSMs with Second-Order Dynamics](https://mlsanity.com/paper/2607.03589/) (2026)
- [Intelligent Communication Mixture-of-Experts Boosted-Medical Image Segmentation Foundation Model](https://mlsanity.com/paper/2510.17684/) (2025)
- [WaveSeg: Enhancing Segmentation Precision via High-Frequency Prior and Mamba-Driven Spectrum Decomposition](https://mlsanity.com/paper/2510.21079/) (2025)
- [Adapting Vision Transformers to Ultra-High Resolution Semantic Segmentation with Relay Tokens](https://mlsanity.com/paper/2601.05927/) (2026)
- [Human-inspired Global-to-Parallel Multi-scale Encoding for Lightweight Vision Models](https://mlsanity.com/paper/2601.08190/) (2026)
- [InfoMamba: An Attention-Free Hybrid Mamba-Transformer Model](https://mlsanity.com/paper/2603.18031/) (2026)
- [Bridging the Geometry Mismatch: Frequency-Aware Anisotropic Serialization for Thin-Structure SSMs](https://mlsanity.com/paper/2603.28503/) (2026)
- [Enhancing Box and Block Test with Computer Vision for Post-Stroke Upper Extremity Motor Evaluation](https://mlsanity.com/paper/2603.29101/) (2026)
- [Bridging the Generalization Gap in Adverse Weather Segmentation: A Training Recipe Perspective](https://mlsanity.com/paper/2605.27962/) (2026)
- [Hierarchical Spatial and Channel Aggregation for Cross-domain Few-shot Segmentation](https://mlsanity.com/paper/2606.24296/) (2026)
- [DishSeg24k: A Large-Scale Benchmark for Food Segmentation with Stochastic Expert Decoding](https://mlsanity.com/paper/2607.23070/) (2026)
- [TRNet: Learning with Topographic Priors for VHR Paddy Rice Mapping](https://mlsanity.com/paper/2608.04154/) (2026)
- [Generative Semantic Segmentation via an Observable Semantic-Image Interface and Hierarchical Generator Evidence Alignment](https://mlsanity.com/paper/2608.11537/) (2026)
- [Contextrast++: Robust Multi-Scale Contextual Contrastive Learning for Semantic Segmentation](https://mlsanity.com/paper/2608.22679/) (2026)

## Abstract (arXiv)

High-quality semantic segmentation relies on three key capabilities: global context modeling, local detail encoding, and multi-scale feature extraction. However, recent methods struggle to possess all these capabilities simultaneously. Hence, we aim to empower segmentation networks to simultaneously carry out efficient global context modeling, high-quality local detail encoding, and rich multi-scale feature representation for varying input resolutions. In this paper, we introduce SegMAN, a novel linear-time model comprising a hybrid feature encoder dubbed SegMAN Encoder, and a decoder based on state space models. Specifically, the SegMAN Encoder synergistically integrates sliding local attention with dynamic state space models, enabling highly efficient global context modeling while preserving fine-grained local details. Meanwhile, the MMSCopE module in our decoder enhances multi-scale context feature extraction and adaptively scales with the input resolution. Our SegMAN-B Encoder achieves 85.1% ImageNet-1k accuracy (+1.5% over VMamba-S with fewer parameters). When paired with our decoder, the full SegMAN-B model achieves 52.6% mIoU on ADE20K (+1.6% over SegNeXt-L with 15% fewer GFLOPs), 83.8% mIoU on Cityscapes (+2.1% over SegFormer-B3 with half the GFLOPs), and 1.6% higher mIoU than VWFormer-B3 on COCO-Stuff with lower GFLOPs. Our code is available at https://github.com/yunxiangfu2001/SegMAN.