# How is Visual Attention Influenced by Text Guidance? Database and Model arXiv: https://arxiv.org/abs/2404.07537 ยท HTML: https://mlsanity.com/paper/2404.07537/ ## Key facts - Cited by 7 papers in the map (data through 14 Oct 2026). - Builds on 33 of its 81 references that are in the map. - First posted on arXiv 11 Apr 2024. ## Builds on (most cited first, 30 of 33) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021) - [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014) - [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016) - [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022) - [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015) - [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021) - [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015) - [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023) - [GLIGEN: Open-Set Grounded Text-to-Image Generation](https://mlsanity.com/paper/2301.07093/) (2023) - [Scaling up GANs for Text-to-Image Synthesis](https://mlsanity.com/paper/2303.05511/) (2023) - [What do different evaluation metrics tell us about saliency models?](https://mlsanity.com/paper/1604.03605/) (2016) - [Predicting Human Eye Fixations via an LSTM-based Saliency Attentive Model](https://mlsanity.com/paper/1611.09571/) (2016) - [Towards Open-ended Visual Quality Comparison](https://mlsanity.com/paper/2402.16641/) (2024) - [SalGAN: Visual Saliency Prediction with Generative Adversarial Networks](https://mlsanity.com/paper/1701.01081/) (2017) - [Deep Gaze I: Boosting Saliency Prediction with Feature Maps Trained on ImageNet](https://mlsanity.com/paper/1411.1045/) (2014) - [Review of Visual Saliency Detection with Comprehensive Information](https://mlsanity.com/paper/1803.03391/) (2018) - [A Deep Multi-Level Network for Saliency Prediction](https://mlsanity.com/paper/1609.01064/) (2016) - [CAT2000: A Large Scale Fixation Dataset for Boosting Saliency Research](https://mlsanity.com/paper/1505.03581/) (2015) - [DeepFix: A Fully Convolutional Neural Network for predicting Human Eye Fixations](https://mlsanity.com/paper/1510.02927/) (2015) - [End-to-End Human-Gaze-Target Detection with Transformers](https://mlsanity.com/paper/2203.10433/) (2022) - [TranSalNet: Towards perceptually relevant visual saliency prediction](https://mlsanity.com/paper/2110.03593/) (2021) - [Perceptual Video Quality Assessment: A Survey](https://mlsanity.com/paper/2402.03413/) (2024) - [Confusing Image Quality Assessment: Towards Better Augmented Reality Experience](https://mlsanity.com/paper/2204.04900/) (2022) - [TempSAL -- Uncovering Temporal Information for Deep Saliency Prediction](https://mlsanity.com/paper/2301.02315/) (2023) - [How is Gaze Influenced by Image Transformations? Dataset and Model](https://mlsanity.com/paper/1905.06803/) (2019) - [A Dilated Inception Network for Visual Saliency Prediction](https://mlsanity.com/paper/1904.03571/) (2019) - [End-to-End Saliency Mapping via Probability Distribution Prediction](https://mlsanity.com/paper/1804.01793/) (2018) - [Saliency in Augmented Reality](https://mlsanity.com/paper/2204.08308/) (2022) - [Progressive Self-Guided Loss for Salient Object Detection](https://mlsanity.com/paper/2101.02412/) (2021) ## Built on by (most cited first, 7 of 7) - [Quality Assessment for AI Generated Images with Instruction Tuning](https://mlsanity.com/paper/2405.07346/) (2024) - [How Does Audio Influence Visual Attention in Omnidirectional Videos? Database and Model](https://mlsanity.com/paper/2408.05411/) (2024) - [AGAV-Rater: Adapting Large Multimodal Model for AI-Generated Audio-Visual Quality Assessment](https://mlsanity.com/paper/2501.18314/) (2025) - [MVBIND: Self-Supervised Music Recommendation For Videos Via Embedding Space Binding](https://mlsanity.com/paper/2405.09286/) (2024) - [Quality-guided Skin Tone Enhancement for Portrait Photography](https://mlsanity.com/paper/2406.15848/) (2024) - [TSalV360: A Method and Dataset for Text-driven Saliency Detection in 360-Degrees Videos](https://mlsanity.com/paper/2509.26208/) (2025) - [TDSal: Task-Based Top-Down Saliency Prediction Model](https://mlsanity.com/paper/2607.09827/) (2026) ## Abstract (arXiv) The analysis and prediction of visual attention have long been crucial tasks in the fields of computer vision and image processing. In practical applications, images are generally accompanied by various text descriptions, however, few studies have explored the influence of text descriptions on visual attention, let alone developed visual saliency prediction models considering text guidance. In this paper, we conduct a comprehensive study on text-guided image saliency (TIS) from both subjective and objective perspectives. Specifically, we construct a TIS database named SJTU-TIS, which includes 1200 text-image pairs and the corresponding collected eye-tracking data. Based on the established SJTU-TIS database, we analyze the influence of various text descriptions on visual attention. Then, to facilitate the development of saliency prediction models considering text influence, we construct a benchmark for the established SJTU-TIS database using state-of-the-art saliency models. Finally, considering the effect of text descriptions on visual attention, while most existing saliency models ignore this impact, we further propose a text-guided saliency (TGSal) prediction model, which extracts and integrates both image features and text features to predict the image saliency under various text-description conditions. Our proposed model significantly outperforms the state-of-the-art saliency models on both the SJTU-TIS database and the pure image saliency databases in terms of various evaluation metrics. The SJTU-TIS database and the code of the proposed TGSal model will be released at: https://github.com/IntMeGroup/TGSal.