# Visual Attention Graph

arXiv: https://arxiv.org/abs/2503.08531 · HTML: https://mlsanity.com/paper/2503.08531/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 101 references that are in the map.
- First posted on arXiv 11 Mar 2025.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [Dual Attention Network for Scene Segmentation](https://mlsanity.com/paper/1809.02983/) (2018)
- [Attention Mechanisms in Computer Vision: A Survey](https://mlsanity.com/paper/2111.07624/) (2021)
- [Deep Visual Attention Prediction](https://mlsanity.com/paper/1705.02544/) (2017)
- [Deep Gaze I: Boosting Saliency Prediction with Feature Maps Trained on ImageNet](https://mlsanity.com/paper/1411.1045/) (2014)
- [DeepGaze IIE: Calibrated prediction in and out-of-domain for state-of-the-art saliency modeling](https://mlsanity.com/paper/2105.12441/) (2021)
- [DeepGaze II: Reading fixations from deep features trained on object recognition](https://mlsanity.com/paper/1610.01563/) (2016)
- [A Deep Multi-Level Network for Saliency Prediction](https://mlsanity.com/paper/1609.01064/) (2016)
- [CAT2000: A Large Scale Fixation Dataset for Boosting Saliency Research](https://mlsanity.com/paper/1505.03581/) (2015)
- [DeepFix: A Fully Convolutional Neural Network for predicting Human Eye Fixations](https://mlsanity.com/paper/1510.02927/) (2015)
- [Follow My Eye: Using Gaze to Supervise Computer-Aided Diagnosis](https://mlsanity.com/paper/2204.02976/) (2022)
- [PathGAN: Visual Scanpath Prediction with Generative Adversarial Networks](https://mlsanity.com/paper/1809.00567/) (2018)
- [SaltiNet: Scan-path Prediction on 360 Degree Images using Saliency Volumes](https://mlsanity.com/paper/1707.03123/) (2017)
- [COCONut: Modernizing COCO Segmentation](https://mlsanity.com/paper/2404.08639/) (2024)
- [State-of-the-Art in Human Scanpath Prediction](https://mlsanity.com/paper/2102.12239/) (2021)
- [Mining Gaze for Contrastive Learning toward Computer-Assisted Diagnosis](https://mlsanity.com/paper/2312.06069/) (2023)
- [Detection of ADHD based on Eye Movements during Natural Viewing](https://mlsanity.com/paper/2207.01377/) (2022)
- [Salient Structure Detection by Context-Guided Visual Search](https://mlsanity.com/paper/1505.04364/) (2015)
- [A Probabilistic Time-Evolving Approach to Scanpath Prediction](https://mlsanity.com/paper/2204.09404/) (2022)
- [Vanishing point attracts gaze in free-viewing and visual search tasks](https://mlsanity.com/paper/1512.01722/) (2015)

## Abstract (arXiv)

Visual attention plays a critical role when our visual system executes active visual tasks by interacting with the physical scene. However, how to encode the visual object relationship in the psychological world of our brain deserves to be explored. In the field of computer vision, predicting visual fixations or scanpaths is a usual way to explore the visual attention and behaviors of human observers when viewing a scene. Most existing methods encode visual attention using individual fixations or scanpaths based on the raw gaze shift data collected from human observers. This may not capture the common attention pattern well, because without considering the semantic information of the viewed scene, raw gaze shift data alone contain high inter- and intra-observer variability. To address this issue, we propose a new attention representation, called Attention Graph, to simultaneously code the visual saliency and scanpath in a graph-based representation and better reveal the common attention behavior of human observers. In the attention graph, the semantic-based scanpath is defined by the path on the graph, while saliency of objects can be obtained by computing fixation density on each node. Systemic experiments demonstrate that the proposed attention graph combined with our new evaluation metrics provides a better benchmark for evaluating attention prediction methods. Meanwhile, extra experiments demonstrate the promising potentials of the proposed attention graph in assessing human cognitive states, such as autism spectrum disorder screening and age classification.