Sketch-based Creativity Support Tools using Deep Learning
Not on the map. Why?
Abstract (arXiv)
Sketching is a natural and effective visual communication medium commonly used in creative processes. Recent developments in deep-learning models drastically improved machines' ability in understanding and generating visual content. An exciting area of development explores deep-learning approaches used to model human sketches, opening opportunities for creative applications. This chapter describes three fundamental steps in developing deep-learning-driven creativity support tools that consumes and generates sketches: 1) a data collection effort that generated a new paired dataset between sketches and mobile user interfaces; 2) a sketch-based user interface retrieval system adapted from state-of-the-art computer vision techniques; and, 3) a conversational sketching system that supports the novel interaction of a natural-language-based sketch/critique authoring process. In this chapter, we survey relevant prior work in both the deep-learning and human-computer-interaction communities, document the data collection process and the systems' architectures in detail, present qualitative and quantitative results, and paint the landscape of several future research directions in this exciting area.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 65 of its 312 references that are in the map. First posted on arXiv 19 Nov 2021.
Builds on
65 of 312 references.
References · 65
- 2017Attention Is All You Needcited by 60,361
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
- 2014Microsoft COCO: Common Objects in Contextcited by 17,608
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
- 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
- 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
- 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
- 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
- 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
- 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
- 2014Deep Learning Face Attributes in the Wildcited by 3,693
- 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,988
38 more
- 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
- 2016Improved Techniques for Training GANscited by 2,970
- 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,849
- 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
- 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,626
- 2015VQA: Visual Question Answeringcited by 2,506
- 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
- 2017Improved Training of Wasserstein GANscited by 2,235
- 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,162
- 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,054
- 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
- 2016Adversarial examples in the physical worldcited by 1,769
- 2018Spectral Normalization for Generative Adversarial Networkscited by 1,636
- 2018Demystifying MMD GANscited by 1,293
- 2018Model Cards for Model Reportingcited by 1,080
- 2014Show and Tell: A Neural Image Caption Generatorcited by 879
- 2018Assessing Generative Models via Precision and Recallcited by 406
- 2019On the Utility of Learning about Humans for Human-AI Coordinationcited by 349
- 2016Visual Relationship Detection with Language Priorscited by 334
- 2017A Neural Representation of Sketch Drawingscited by 313
- 2017Are GANs Created Equal? A Large-Scale Studycited by 267
- 2017Recent Trends in Deep Learning Based Natural Language Processingcited by 263
- 2015YFCC100M: The New Data in Multimedia Researchcited by 235
- 2018A Note on the Inception Scorecited by 226
- 2018Pros and Cons of GAN Evaluation Measurescited by 224
- 2015Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networkscited by 214
- 2020Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systemscited by 139
- 2015Character-Aware Neural Language Modelscited by 132
- 2019Unifying Human and Statistical Evaluation for Natural Language Generationcited by 127
- 2017BEGAN: Boundary Equilibrium Generative Adversarial Networkscited by 121
- 2014The Rise of Social Botscited by 96
- 2019Photo-Sketching: Inferring Contour Drawings from Imagescited by 68
- 2020SketchyCOCO: Image Generation from Freehand Scene Sketchescited by 67
- 2019HYPE: A Benchmark for Human eYe Perceptual Evaluation of Generative Modelscited by 59
- 2017Variational Approaches for Auto-Encoding Generative Adversarial Networkscited by 52
- 2020A Cordial Sync: Going Beyond Marginal Policies for Multi-Agent Embodied Taskscited by 52
- 2020Conceptual Metaphors Impact Perceptions of Human-AI Collaborationcited by 47
- 2019Information Maximizing Visual Question Generationcited by 40
50 most cited of 65 All 65 (JSON)
Built on by
Citing papers per year: 2022: 1; 2023: 2.
3 papers.