# Sketch2Saliency: Learning to Detect Salient Objects from Human Drawings

arXiv: https://arxiv.org/abs/2303.11502 · HTML: https://mlsanity.com/paper/2303.11502/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 58 of its 91 references that are in the map.
- First posted on arXiv 20 Mar 2023.

## Builds on (most cited first, 30 of 58)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [Self-Attention Generative Adversarial Networks](https://mlsanity.com/paper/1805.08318/) (2018)
- [Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey](https://mlsanity.com/paper/1902.06162/) (2019)
- [Generating Sequences With Recurrent Neural Networks](https://mlsanity.com/paper/1308.0850/) (2013)
- [Training Deep Neural Networks on Noisy Labels with Bootstrapping](https://mlsanity.com/paper/1412.6596/) (2014)
- [A Neural Representation of Sketch Drawings](https://mlsanity.com/paper/1704.03477/) (2017)
- [Self-supervised Equivariant Attention Mechanism for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2004.04581/) (2020)
- [Object Region Mining with Adversarial Erasing: A Simple Classification to Semantic Segmentation Approach](https://mlsanity.com/paper/1703.08448/) (2017)
- [The Secrets of Salient Object Segmentation](https://mlsanity.com/paper/1406.2807/) (2014)
- [Visual Saliency Based on Multiscale Deep Features](https://mlsanity.com/paper/1503.08663/) (2015)
- [Salient Object Detection in the Deep Learning Era: An In-Depth Survey](https://mlsanity.com/paper/1904.09146/) (2019)
- [Reverse Attention for Salient Object Detection](https://mlsanity.com/paper/1807.09940/) (2018)
- [SCOPS: Self-Supervised Co-Part Segmentation](https://mlsanity.com/paper/1905.01298/) (2019)
- [Weakly-Supervised Salient Object Detection via Scribble Annotations](https://mlsanity.com/paper/2003.07685/) (2020)
- [Salient Object Detection: A Discriminative Regional Feature Integration Approach](https://mlsanity.com/paper/1410.5926/) (2014)
- [Sketchformer: Transformer-based Representation for Sketched Structure](https://mlsanity.com/paper/2002.10381/) (2020)
- [A Zero-Shot Framework for Sketch-based Image Retrieval](https://mlsanity.com/paper/1807.11724/) (2018)
- [SketchyCOCO: Image Generation from Freehand Scene Sketches](https://mlsanity.com/paper/2003.02683/) (2020)
- [Deep Learning for Free-Hand Sketch: A Survey](https://mlsanity.com/paper/2001.02600/) (2020)
- [Sketch Less for More: On-the-Fly Fine-Grained Sketch Based Image Retrieval](https://mlsanity.com/paper/2002.10310/) (2020)
- [Semantically Tied Paired Cycle Consistency for Zero-Shot Sketch-based Image Retrieval](https://mlsanity.com/paper/1903.03372/) (2019)
- [Joint Learning of Saliency Detection and Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/1909.04161/) (2019)
- [Structure-Consistent Weakly Supervised Salient Object Detection with Local Saliency Coherence](https://mlsanity.com/paper/2012.04404/) (2020)
- [Sketch2Model: View-Aware 3D Modeling from Single Free-Hand Sketches](https://mlsanity.com/paper/2105.06663/) (2021)
- [Interactive Sketch & Fill: Multiclass Sketch-to-Image Translation](https://mlsanity.com/paper/1909.11081/) (2019)

## Built on by (most cited first, 17 of 17)

- [CLIP for All Things Zero-Shot Sketch-Based Image Retrieval, Fine-Grained or Not](https://mlsanity.com/paper/2303.13440/) (2023)
- [RT-Sketch: Goal-Conditioned Imitation Learning from Hand-Drawn Sketches](https://mlsanity.com/paper/2403.02709/) (2024)
- [It's All About Your Sketch: Democratising Sketch Control in Diffusion Models](https://mlsanity.com/paper/2403.07234/) (2024)
- [What Can Human Sketches Do for Object Detection?](https://mlsanity.com/paper/2303.15149/) (2023)
- [Exploiting Unlabelled Photos for Stronger Fine-Grained SBIR](https://mlsanity.com/paper/2303.13779/) (2023)
- [How to Handle Sketch-Abstraction in Sketch-Based Image Retrieval?](https://mlsanity.com/paper/2403.07203/) (2024)
- [Sketch-to-Skill: Bootstrapping Robot Learning with Human Drawn Trajectory Sketches](https://mlsanity.com/paper/2503.11918/) (2025)
- [SketchFusion: Learning Universal Sketch Features through Fusing Foundation Models](https://mlsanity.com/paper/2503.14129/) (2025)
- [SketchMind: A Multi-Agent Cognitive Framework for Assessing Student-Drawn Scientific Sketches](https://mlsanity.com/paper/2507.22904/) (2025)
- [Stroke of Surprise: Progressive Semantic Illusions in Vector Sketching](https://mlsanity.com/paper/2602.12280/) (2026)
- [Multi-modal Generation via Cross-Modal In-Context Learning](https://mlsanity.com/paper/2405.18304/) (2024)
- [Sketch Down the FLOPs: Towards Efficient Networks for Human Sketch](https://mlsanity.com/paper/2505.23763/) (2025)
- [SketchYourSeg: Mask-Free Subjective Image Segmentation via Freehand Sketches](https://mlsanity.com/paper/2501.16022/) (2025)
- [SketchDeco: Training-Free Latent Composition for Precise Sketch Colourisation](https://mlsanity.com/paper/2405.18716/) (2024)
- [Breaking the Box: Enhancing Remote Sensing Image Segmentation with Freehand Sketches](https://mlsanity.com/paper/2503.12191/) (2025)
- [Doodle Your Keypoints: Sketch-Based Few-Shot Keypoint Detection](https://mlsanity.com/paper/2507.07994/) (2025)
- [Focus Where It Counts: A Salience-Driven Vision-Language Model for Low Vision Assistance](https://mlsanity.com/paper/2608.28218/) (2026)

## Abstract (arXiv)

Human sketch has already proved its worth in various visual understanding tasks (e.g., retrieval, segmentation, image-captioning, etc). In this paper, we reveal a new trait of sketches - that they are also salient. This is intuitive as sketching is a natural attentive process at its core. More specifically, we aim to study how sketches can be used as a weak label to detect salient objects present in an image. To this end, we propose a novel method that emphasises on how "salient object" could be explained by hand-drawn sketches. To accomplish this, we introduce a photo-to-sketch generation model that aims to generate sequential sketch coordinates corresponding to a given visual photo through a 2D attention mechanism. Attention maps accumulated across the time steps give rise to salient regions in the process. Extensive quantitative and qualitative experiments prove our hypothesis and delineate how our sketch-based saliency detection model gives a competitive performance compared to the state-of-the-art.