# Class-Agnostic Visio-Temporal Scene Sketch Semantic Segmentation

arXiv: https://arxiv.org/abs/2410.00266 · HTML: https://mlsanity.com/paper/2410.00266/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 43 references that are in the map.
- First posted on arXiv 30 Sep 2024.

## Builds on (most cited first, 25 of 25)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation](https://mlsanity.com/paper/1511.00561/) (2015)
- [DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection](https://mlsanity.com/paper/2203.03605/) (2022)
- [YOLOX: Exceeding YOLO Series in 2021](https://mlsanity.com/paper/2107.08430/) (2021)
- [COCO-Stuff: Thing and Stuff Classes in Context](https://mlsanity.com/paper/1612.03716/) (2016)
- [Learning Deconvolution Network for Semantic Segmentation](https://mlsanity.com/paper/1505.04366/) (2015)
- [End-to-End Semi-Supervised Object Detection with Soft Teacher](https://mlsanity.com/paper/2106.09018/) (2021)
- [A Neural Representation of Sketch Drawings](https://mlsanity.com/paper/1704.03477/) (2017)
- [Unbiased Mean Teacher for Cross-domain Object Detection](https://mlsanity.com/paper/2003.00707/) (2020)
- [Sketchformer: Transformer-based Representation for Sketched Structure](https://mlsanity.com/paper/2002.10381/) (2020)
- [SketchyCOCO: Image Generation from Freehand Scene Sketches](https://mlsanity.com/paper/2003.02683/) (2020)
- [YOLOv6 v3.0: A Full-Scale Reloading](https://mlsanity.com/paper/2301.05586/) (2023)
- [FS-COCO: Towards Understanding of Freehand Sketches of Common Objects in Context](https://mlsanity.com/paper/2203.02113/) (2022)
- [Decoupled Adaptation for Cross-Domain Object Detection](https://mlsanity.com/paper/2110.02578/) (2021)
- [Multi-Graph Transformer for Free-Hand Sketch Recognition](https://mlsanity.com/paper/1912.11258/) (2019)
- [SketchyScene: Richly-Annotated Scene Sketches](https://mlsanity.com/paper/1808.02473/) (2018)
- [Fast Sketch Segmentation and Labeling with Deep Learning](https://mlsanity.com/paper/1807.11847/) (2018)
- [Stroke-based sketched symbol reconstruction and segmentation](https://mlsanity.com/paper/1901.03427/) (2019)
- [Open Vocabulary Semantic Scene Sketch Understanding](https://mlsanity.com/paper/2312.12463/) (2023)
- [Domain-Adaptive Self-Supervised Pre-Training for Face & Body Detection in Drawings](https://mlsanity.com/paper/2211.10641/) (2022)

## Built on by (most cited first, 2 of 2)

- [LASA: A Weak Supervision Method for Open-Vocabulary Scene Sketch Semantic Segmentation](https://mlsanity.com/paper/2606.11837/) (2026)
- [ProgressNet: Sketching and Prompting with a Frozen Text-to-Image Model](https://mlsanity.com/paper/2610.03512/) (2026)

## Abstract (arXiv)

Scene sketch semantic segmentation is a crucial task for various applications including sketch-to-image retrieval and scene understanding. Existing sketch segmentation methods treat sketches as bitmap images, leading to the loss of temporal order among strokes due to the shift from vector to image format. Moreover, these methods struggle to segment objects from categories absent in the training data. In this paper, we propose a Class-Agnostic Visio-Temporal Network (CAVT) for scene sketch semantic segmentation. CAVT employs a class-agnostic object detector to detect individual objects in a scene and groups the strokes of instances through its post-processing module. This is the first approach that performs segmentation at both the instance and stroke levels within scene sketches. Furthermore, there is a lack of free-hand scene sketch datasets with both instance and stroke-level class annotations. To fill this gap, we collected the largest Free-hand Instance- and Stroke-level Scene Sketch Dataset (FrISS) that contains 1K scene sketches and covers 403 object classes with dense annotations. Extensive experiments on FrISS and other datasets demonstrate the superior performance of our method over state-of-the-art scene sketch segmentation models. The code and dataset will be made public after acceptance.