# SUM: Saliency Unification through Mamba for Visual Attention Modeling

arXiv: https://arxiv.org/abs/2406.17815 · HTML: https://mlsanity.com/paper/2406.17815/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Vision Mamba backbones, one of 10 directions in State-space sequence models (S4, Mamba).
- Builds on 38 of its 70 references that are in the map.
- First posted on arXiv 25 Jun 2024.

## Placed in

- State-space sequence models (S4, Mamba) / [Vision Mamba backbones](https://mlsanity.com/ssm/vision-mamba-backbones/)

## Builds on (most cited first, 30 of 38)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model](https://mlsanity.com/paper/2401.09417/) (2024)
- [VMamba: Visual State Space Model](https://mlsanity.com/paper/2401.10166/) (2024)
- [Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning](https://mlsanity.com/paper/1702.03118/) (2017)
- [A Survey on Visual Transformer](https://mlsanity.com/paper/2012.12556/) (2020)
- [U-Mamba: Enhancing Long-range Dependency for Biomedical Image Segmentation](https://mlsanity.com/paper/2401.04722/) (2024)
- [VM-UNet: Vision Mamba UNet for Medical Image Segmentation](https://mlsanity.com/paper/2402.02491/) (2024)
- [SegMamba: Long-range Sequential Modeling Mamba For 3D Medical Image Segmentation](https://mlsanity.com/paper/2401.13560/) (2024)
- [What do different evaluation metrics tell us about saliency models?](https://mlsanity.com/paper/1604.03605/) (2016)
- [EfficientVMamba: Atrous Selective Scan for Light Weight Visual Mamba](https://mlsanity.com/paper/2403.09977/) (2024)
- [PlainMamba: Improving Non-Hierarchical Mamba in Visual Recognition](https://mlsanity.com/paper/2403.17695/) (2024)
- [Video Mamba Suite: State Space Model as a Versatile Alternative for Video Understanding](https://mlsanity.com/paper/2403.09626/) (2024)
- [Predicting Human Eye Fixations via an LSTM-based Saliency Attentive Model](https://mlsanity.com/paper/1611.09571/) (2016)
- [Unified Image and Video Saliency Modeling](https://mlsanity.com/paper/2003.05477/) (2020)
- [Deep Visual Attention Prediction](https://mlsanity.com/paper/1705.02544/) (2017)
- [SalGAN: Visual Saliency Prediction with Generative Adversarial Networks](https://mlsanity.com/paper/1701.01081/) (2017)
- [Deep Gaze I: Boosting Saliency Prediction with Feature Maps Trained on ImageNet](https://mlsanity.com/paper/1411.1045/) (2014)
- [MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts](https://mlsanity.com/paper/2401.04081/) (2024)
- [DeepGaze IIE: Calibrated prediction in and out-of-domain for state-of-the-art saliency modeling](https://mlsanity.com/paper/2105.12441/) (2021)
- [DeepGaze II: Reading fixations from deep features trained on object recognition](https://mlsanity.com/paper/1610.01563/) (2016)
- [LightM-UNet: Mamba Assists in Lightweight UNet for Medical Image Segmentation](https://mlsanity.com/paper/2403.05246/) (2024)
- [CAT2000: A Large Scale Fixation Dataset for Boosting Saliency Research](https://mlsanity.com/paper/1505.03581/) (2015)
- [EML-NET:An Expandable Multi-Layer NETwork for Saliency Prediction](https://mlsanity.com/paper/1805.01047/) (2018)
- [DeepFix: A Fully Convolutional Neural Network for predicting Human Eye Fixations](https://mlsanity.com/paper/1510.02927/) (2015)
- [Contextual Encoder-Decoder Network for Visual Saliency Prediction](https://mlsanity.com/paper/1902.06634/) (2019)
- [TranSalNet: Towards perceptually relevant visual saliency prediction](https://mlsanity.com/paper/2110.03593/) (2021)
- [Predicting Visual Importance Across Graphic Design Types](https://mlsanity.com/paper/2008.02912/) (2020)

## Built on by (most cited first, 12 of 12)

- [A Survey of Mamba](https://mlsanity.com/paper/2408.01129/) (2024)
- [DiffEye: Diffusion-Based Continuous Eye-Tracking Data Generation Conditioned on Natural Images](https://mlsanity.com/paper/2509.16767/) (2025)
- [DTFSal: Audio-Visual Dynamic Token Fusion for Video Saliency Prediction](https://mlsanity.com/paper/2504.10070/) (2025)
- [Attend to Anything: Foundation Model for Unified Human Attention Modeling](https://mlsanity.com/paper/2606.03540/) (2026)
- [Efficient Malicious UAV Detection Using Autoencoder-TSMamba Integration](https://mlsanity.com/paper/2505.10585/) (2025)
- [PRE-MAP: Personalized Reinforced Eye-tracking Multimodal LLM for High-Resolution Multi-Attribute Point Prediction](https://mlsanity.com/paper/2507.19213/) (2025)
- [EgoCampus: Egocentric Pedestrian Eye Gaze Model and Dataset](https://mlsanity.com/paper/2512.07668/) (2025)
- [ObjectVisA-120: Object-based Visual Attention Prediction in Interactive Street-crossing Environments](https://mlsanity.com/paper/2601.13218/) (2026)
- [SIMON: Saliency-aware Integrative Multi-view Object-centric Neural Decoding](https://mlsanity.com/paper/2605.00401/) (2026)
- [ST-DiffEye: Diffusion-based Continuous Gaze Generation via Joint Scanpath-Trajectory Modeling](https://mlsanity.com/paper/2606.15486/) (2026)
- [Fast Human Attention Prediction for Fixation-guided Active Perception in Autonomous Navigation](https://mlsanity.com/paper/2606.20491/) (2026)
- [OpenVAM: Open-World Visual Attention Modeling with VLMs](https://mlsanity.com/paper/2609.31364/) (2026)

## Abstract (arXiv)

Visual attention modeling, important for interpreting and prioritizing visual stimuli, plays a significant role in applications such as marketing, multimedia, and robotics. Traditional saliency prediction models, especially those based on Convolutional Neural Networks (CNNs) or Transformers, achieve notable success by leveraging large-scale annotated datasets. However, the current state-of-the-art (SOTA) models that use Transformers are computationally expensive. Additionally, separate models are often required for each image type, lacking a unified approach. In this paper, we propose Saliency Unification through Mamba (SUM), a novel approach that integrates the efficient long-range dependency modeling of Mamba with U-Net to provide a unified model for diverse image types. Using a novel Conditional Visual State Space (C-VSS) block, SUM dynamically adapts to various image types, including natural scenes, web pages, and commercial imagery, ensuring universal applicability across different data types. Our comprehensive evaluations across five benchmarks demonstrate that SUM seamlessly adapts to different visual characteristics and consistently outperforms existing models. These results position SUM as a versatile and powerful tool for advancing visual attention modeling, offering a robust solution universally applicable across different types of visual content.