# Predicting Visual Attention in Graphic Design Documents

arXiv: https://arxiv.org/abs/2407.02439 · HTML: https://mlsanity.com/paper/2407.02439/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 74 references that are in the map.
- First posted on arXiv 2 Jul 2024.

## Builds on (most cited first, 24 of 24)

- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [UNet++: A Nested U-Net Architecture for Medical Image Segmentation](https://mlsanity.com/paper/1807.10165/) (2018)
- [Visual Saliency Based on Multiscale Deep Features](https://mlsanity.com/paper/1503.08663/) (2015)
- [What do different evaluation metrics tell us about saliency models?](https://mlsanity.com/paper/1604.03605/) (2016)
- [TextSnake: A Flexible Representation for Detecting Text of Arbitrary Shapes](https://mlsanity.com/paper/1807.01544/) (2018)
- [Predicting Human Eye Fixations via an LSTM-based Saliency Attentive Model](https://mlsanity.com/paper/1611.09571/) (2016)
- [Deep Visual Attention Prediction](https://mlsanity.com/paper/1705.02544/) (2017)
- [Revisiting Video Saliency: A Large-scale Benchmark and a New Model](https://mlsanity.com/paper/1801.07424/) (2018)
- [DeepGaze IIE: Calibrated prediction in and out-of-domain for state-of-the-art saliency modeling](https://mlsanity.com/paper/2105.12441/) (2021)
- [DeepGaze II: Reading fixations from deep features trained on object recognition](https://mlsanity.com/paper/1610.01563/) (2016)
- [Predicting Goal-directed Human Attention Using Inverse Reinforcement Learning](https://mlsanity.com/paper/2005.14310/) (2020)
- [EML-NET:An Expandable Multi-Layer NETwork for Saliency Prediction](https://mlsanity.com/paper/1805.01047/) (2018)
- [Predicting Visual Importance Across Graphic Design Types](https://mlsanity.com/paper/2008.02912/) (2020)
- [A Deep Spatial Contextual Long-term Recurrent Convolutional Network for Saliency Detection](https://mlsanity.com/paper/1610.01708/) (2016)
- [Top-down Visual Saliency Guided by Captions](https://mlsanity.com/paper/1612.07360/) (2016)
- [Learning Visual Importance for Graphic Designs and Data Visualizations](https://mlsanity.com/paper/1708.02660/) (2017)
- [A Dilated Inception Network for Visual Saliency Prediction](https://mlsanity.com/paper/1904.03571/) (2019)
- [Understanding and Visualizing Deep Visual Saliency Models](https://mlsanity.com/paper/1903.02501/) (2019)
- [Understanding Visual Saliency in Mobile User Interfaces](https://mlsanity.com/paper/2101.09176/) (2021)
- [Saliency Prediction for Mobile User Interfaces](https://mlsanity.com/paper/1711.03726/) (2017)
- [An Element Sensitive Saliency Model with Position Prior Learning for Web Pages](https://mlsanity.com/paper/1804.10361/) (2018)

## Built on by (most cited first, 3 of 3)

- [UICrit: Enhancing Automated Design Evaluation with a UICritique Dataset](https://mlsanity.com/paper/2407.08850/) (2024)
- [GazeXplain: Learning to Predict Natural Language Explanations of Visual Scanpaths](https://mlsanity.com/paper/2408.02788/) (2024)
- [Offline Evaluation of Set-Based Text-to-Image Generation](https://mlsanity.com/paper/2410.17331/) (2024)

## Abstract (arXiv)

We present a model for predicting visual attention during the free viewing of graphic design documents. While existing works on this topic have aimed at predicting static saliency of graphic designs, our work is the first attempt to predict both spatial attention and dynamic temporal order in which the document regions are fixated by gaze using a deep learning based model. We propose a two-stage model for predicting dynamic attention on such documents, with webpages being our primary choice of document design for demonstration. In the first stage, we predict the saliency maps for each of the document components (e.g. logos, banners, texts, etc. for webpages) conditioned on the type of document layout. These component saliency maps are then jointly used to predict the overall document saliency. In the second stage, we use these layout-specific component saliency maps as the state representation for an inverse reinforcement learning model of fixation scanpath prediction during document viewing. To test our model, we collected a new dataset consisting of eye movements from 41 people freely viewing 450 webpages (the largest dataset of its kind). Experimental results show that our model outperforms existing models in both saliency and scanpath prediction for webpages, and also generalizes very well to other graphic design documents such as comics, posters, mobile UIs, etc. and natural images.