# Sketch Input Method Editor: A Comprehensive Dataset and Methodology for Systematic Input Recognition

arXiv: https://arxiv.org/abs/2311.18254 · HTML: https://mlsanity.com/paper/2311.18254/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 48 references that are in the map.
- First posted on arXiv 30 Nov 2023.

## Builds on (most cited first, 28 of 28)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Dynamic Graph CNN for Learning on Point Clouds](https://mlsanity.com/paper/1801.07829/) (2018)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [Deep CORAL: Correlation Alignment for Deep Domain Adaptation](https://mlsanity.com/paper/1607.01719/) (2016)
- [Deep Domain Confusion: Maximizing for Domain Invariance](https://mlsanity.com/paper/1412.3474/) (2014)
- [DeepGCNs: Can GCNs Go as Deep as CNNs?](https://mlsanity.com/paper/1904.03751/) (2019)
- [A Neural Representation of Sketch Drawings](https://mlsanity.com/paper/1704.03477/) (2017)
- [Few-Shot Class-Incremental Learning](https://mlsanity.com/paper/2004.10956/) (2020)
- [Few-Shot Incremental Learning with Continually Evolved Classifiers](https://mlsanity.com/paper/2104.03047/) (2021)
- [Correlation Alignment for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1612.01939/) (2016)
- [Self-Promoted Prototype Refinement for Few-Shot Class-Incremental Learning](https://mlsanity.com/paper/2107.08918/) (2021)
- [Constrained Few-shot Class-incremental Learning](https://mlsanity.com/paper/2203.16588/) (2022)
- [Few-Shot Class-Incremental Learning by Sampling Multi-Phase Tasks](https://mlsanity.com/paper/2203.17030/) (2022)
- [Deep Learning for Free-Hand Sketch: A Survey](https://mlsanity.com/paper/2001.02600/) (2020)
- [Sketch-based 3D Shape Retrieval using Convolutional Neural Networks](https://mlsanity.com/paper/1504.03504/) (2015)
- [Domain-Adaptive Few-Shot Learning](https://mlsanity.com/paper/2003.08626/) (2020)
- [SketchMate: Deep Hashing for Million-Scale Human Sketch Retrieval](https://mlsanity.com/paper/1804.01401/) (2018)
- [Multi-Graph Transformer for Free-Hand Sketch Recognition](https://mlsanity.com/paper/1912.11258/) (2019)
- [Cloud2Curve: Generation and Vectorization of Parametric Sketches](https://mlsanity.com/paper/2103.15536/) (2021)
- [Universal Perceptual Grouping](https://mlsanity.com/paper/1808.02312/) (2018)
- [Fast Sketch Segmentation and Labeling with Deep Learning](https://mlsanity.com/paper/1807.11847/) (2018)
- [Enabling My Robot To Play Pictionary : Recurrent Neural Networks For Sketch Recognition](https://mlsanity.com/paper/1608.03369/) (2016)
- [SketchParse : Towards Rich Descriptions for Poorly Drawn Sketches using Multi-Task Hierarchical Deep Networks](https://mlsanity.com/paper/1709.01295/) (2017)
- [Object category understanding via eye fixations on freehand sketches](https://mlsanity.com/paper/1703.06554/) (2017)
- [Sequential Dual Deep Learning with Shape and Texture Features for Sketch Recognition](https://mlsanity.com/paper/1708.02716/) (2017)
- [Distribution-Aware Binarization of Neural Networks for Sketch Recognition](https://mlsanity.com/paper/1804.02941/) (2018)
- [Evaluation of Deep Neural Network Domain Adaptation Techniques for Image Recognition](https://mlsanity.com/paper/2109.13420/) (2021)

## Built on by (most cited first, 2 of 2)

- [Enhance Sketch Recognition's Explainability via Semantic Component-Level Parsing](https://mlsanity.com/paper/2312.07875/) (2023)
- [Content-Conditioned Generation of Stylized Free hand Sketches](https://mlsanity.com/paper/2401.04739/) (2024)

## Abstract (arXiv)

With the recent surge in the use of touchscreen devices, free-hand sketching has emerged as a promising modality for human-computer interaction. While previous research has focused on tasks such as recognition, retrieval, and generation of familiar everyday objects, this study aims to create a Sketch Input Method Editor (SketchIME) specifically designed for a professional C4I system. Within this system, sketches are utilized as low-fidelity prototypes for recommending standardized symbols in the creation of comprehensive situation maps. This paper also presents a systematic dataset comprising 374 specialized sketch types, and proposes a simultaneous recognition and segmentation architecture with multilevel supervision between recognition and segmentation to improve performance and enhance interpretability. By incorporating few-shot domain adaptation and class-incremental learning, the network's ability to adapt to new users and extend to new task-specific classes is significantly enhanced. Results from experiments conducted on both the proposed dataset and the SPG dataset illustrate the superior performance of the proposed architecture. Our dataset and code are publicly available at https://github.com/GuangmingZhu/SketchIME.