# Top-down Visual Saliency Guided by Captions

arXiv: https://arxiv.org/abs/1612.07360 · HTML: https://mlsanity.com/paper/1612.07360/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Dec 2016.

## Built on by (most cited first, 22 of 22)

- [Towards Out-Of-Distribution Generalization: A Survey](https://mlsanity.com/paper/2108.13624/) (2021)
- [From Show to Tell: A Survey on Deep Learning-based Image Captioning](https://mlsanity.com/paper/2107.06912/) (2021)
- [DRAMA: Joint Risk Localization and Captioning in Driving](https://mlsanity.com/paper/2209.10767/) (2022)
- [Negative Sample Matters: A Renaissance of Metric Learning for Temporal Grounding](https://mlsanity.com/paper/2109.04872/) (2021)
- [Multi-Modal Knowledge Graph Construction and Application: A Survey](https://mlsanity.com/paper/2202.05786/) (2022)
- [Multimodal Explainable Artificial Intelligence: A Comprehensive Review of Methodological Advances and Future Research Directions](https://mlsanity.com/paper/2306.05731/) (2023)
- [Salient Objects in Clutter](https://mlsanity.com/paper/2105.03053/) (2021)
- [ODAM: Gradient-based instance-specific visual explanations for object detection](https://mlsanity.com/paper/2304.06354/) (2023)
- [Towards Modeling Human Attention from Eye Movements for Neural Source Code Summarization](https://mlsanity.com/paper/2305.09773/) (2023)
- [Scribble-based Boundary-aware Network for Weakly Supervised Salient Object Detection in Remote Sensing Images](https://mlsanity.com/paper/2202.03501/) (2022)
- [Saliency-Aware Class-Agnostic Food Image Segmentation](https://mlsanity.com/paper/2102.06882/) (2021)
- [Understanding and Modeling the Effects of Task and Context on Drivers' Gaze Allocation](https://mlsanity.com/paper/2310.09275/) (2023)
- [Measuring and Predicting Where and When Pathologists Focus their Visual Attention while Grading Whole Slide Images of Cancer](https://mlsanity.com/paper/2508.01668/) (2025)
- [Top-Down Guidance for Learning Object-Centric Representations](https://mlsanity.com/paper/2405.10598/) (2024)
- [Predicting Visual Attention in Graphic Design Documents](https://mlsanity.com/paper/2407.02439/) (2024)
- [Text-Audio-Visual-conditioned Diffusion Model for Video Saliency Prediction](https://mlsanity.com/paper/2504.14267/) (2025)
- [Weakly-Supervised Visual-Textual Grounding with Semantic Prior Refinement](https://mlsanity.com/paper/2305.10913/) (2023)
- [SnapCap: Efficient Snapshot Compressive Video Captioning](https://mlsanity.com/paper/2401.04903/) (2024)
- [Unraveling the Dilemma of AI Errors: Exploring the Effectiveness of Human and Machine Explanations for Large Language Models](https://mlsanity.com/paper/2404.07725/) (2024)
- [Temporal Relevance Analysis for Video Action Models](https://mlsanity.com/paper/2204.11929/) (2022)
- [Relevance-guided Audio Visual Fusion for Video Saliency Prediction](https://mlsanity.com/paper/2411.11454/) (2024)
- [TDSal: Task-Based Top-Down Saliency Prediction Model](https://mlsanity.com/paper/2607.09827/) (2026)

## Abstract (arXiv)

Neural image/video captioning models can generate accurate descriptions, but their internal process of mapping regions to words is a black box and therefore difficult to explain. Top-down neural saliency methods can find important regions given a high-level semantic task such as object classification, but cannot use a natural language sentence as the top-down input for the task. In this paper, we propose Caption-Guided Visual Saliency to expose the region-to-word mapping in modern encoder-decoder networks and demonstrate that it is learned implicitly from caption training data, without any pixel-level annotations. Our approach can produce spatial or spatiotemporal heatmaps for both predicted captions, and for arbitrary query sentences. It recovers saliency without the overhead of introducing explicit attention layers, and can be used to analyze a variety of existing model architectures and improve their design. Evaluation on large-scale video and image datasets demonstrates that our approach achieves comparable captioning performance with existing methods while providing more accurate saliency heatmaps. Our code is available at visionlearninggroup.github.io/caption-guided-saliency/.