# Instance Segmentation of Scene Sketches Using Natural Image Priors

arXiv: https://arxiv.org/abs/2502.09608 · HTML: https://mlsanity.com/paper/2502.09608/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 51 references that are in the map.
- First posted on arXiv 13 Feb 2025.

## Builds on (most cited first, 21 of 21)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks](https://mlsanity.com/paper/2401.14159/) (2024)
- [YOLACT: Real-time Instance Segmentation](https://mlsanity.com/paper/1904.02689/) (2019)
- [Recognize Anything: A Strong Image Tagging Model](https://mlsanity.com/paper/2306.03514/) (2023)
- [Simple Open-Vocabulary Object Detection with Vision Transformers](https://mlsanity.com/paper/2205.06230/) (2022)
- [CLIPasso: Semantically-Aware Object Sketching](https://mlsanity.com/paper/2202.05822/) (2022)
- [FS-COCO: Towards Understanding of Freehand Sketches of Common Objects in Context](https://mlsanity.com/paper/2203.02113/) (2022)
- [Creative Sketch Generation](https://mlsanity.com/paper/2011.10039/) (2020)
- [SketchyScene: Richly-Annotated Scene Sketches](https://mlsanity.com/paper/1808.02473/) (2018)
- [SOLO: A Simple Framework for Instance Segmentation](https://mlsanity.com/paper/2106.15947/) (2021)
- [SketchAgent: Language-Driven Sequential Sketch Generation](https://mlsanity.com/paper/2411.17673/) (2024)
- [One Sketch for All: One-Shot Personalized Sketch Segmentation](https://mlsanity.com/paper/2112.10838/) (2021)
- [ContextSeg: Sketch Semantic Segmentation by Querying the Context with Attention](https://mlsanity.com/paper/2311.16682/) (2023)
- [Stroke-based sketched symbol reconstruction and segmentation](https://mlsanity.com/paper/1901.03427/) (2019)
- [Multi-column Point-CNN for Sketch Segmentation](https://mlsanity.com/paper/1812.11029/) (2018)

## Built on by (most cited first, 4 of 4)

- [Illustrator's Depth: Monocular Layer Index Prediction for Image Decomposition](https://mlsanity.com/paper/2511.17454/) (2025)
- [Creo: From One-Shot Image Generation to Progressive, Co-Creative Ideation](https://mlsanity.com/paper/2604.13956/) (2026)
- [Geometrically Consistent Multi-View Scene Generation from Freehand Sketches](https://mlsanity.com/paper/2604.14302/) (2026)
- [LASA: A Weak Supervision Method for Open-Vocabulary Scene Sketch Semantic Segmentation](https://mlsanity.com/paper/2606.11837/) (2026)

## Abstract (arXiv)

Sketch segmentation involves grouping pixels within a sketch that belong to the same object or instance. It serves as a valuable tool for sketch editing tasks, such as moving, scaling, or removing specific components. While image segmentation models have demonstrated remarkable capabilities in recent years, sketches present unique challenges for these models due to their sparse nature and wide variation in styles. We introduce InkLayer, a method for instance segmentation of raster scene sketches. Our approach adapts state-of-the-art image segmentation and object detection models to the sketch domain by employing class-agnostic fine-tuning and refining segmentation masks using depth cues. Furthermore, our method organizes sketches into sorted layers, where occluded instances are inpainted, enabling advanced sketch editing applications. As existing datasets in this domain lack variation in sketch styles, we construct a synthetic scene sketch segmentation dataset, InkScenes, featuring sketches with diverse brush strokes and varying levels of detail. We use this dataset to demonstrate the robustness of our approach.