# CLIPascene: Scene Sketching with Different Types and Levels of Abstraction

arXiv: https://arxiv.org/abs/2211.17256 · HTML: https://mlsanity.com/paper/2211.17256/

## Key facts

- Cited by 64 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 54 references that are in the map.
- First posted on arXiv 30 Nov 2022.

## Builds on (most cited first, 30 of 36)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [Scaling Autoregressive Models for Content-Rich Text-to-Image Generation](https://mlsanity.com/paper/2206.10789/) (2022)
- [GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks](https://mlsanity.com/paper/1711.02257/) (2017)
- [Resolution-robust Large Mask Inpainting with Fourier Convolutions](https://mlsanity.com/paper/2109.07161/) (2021)
- [Do Vision Transformers See Like Convolutional Neural Networks?](https://mlsanity.com/paper/2108.08810/) (2021)
- [Self-Normalizing Neural Networks](https://mlsanity.com/paper/1706.02515/) (2017)
- [Holistically-Nested Edge Detection](https://mlsanity.com/paper/1504.06375/) (2015)
- [U2-Net: Going Deeper with Nested U-Structure for Salient Object Detection](https://mlsanity.com/paper/2005.09007/) (2020)
- [Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers](https://mlsanity.com/paper/2103.15679/) (2021)
- [A Neural Representation of Sketch Drawings](https://mlsanity.com/paper/1704.03477/) (2017)
- [CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoders](https://mlsanity.com/paper/2106.14843/) (2021)
- [Learning to generate line drawings that convey geometry and semantics](https://mlsanity.com/paper/2203.12691/) (2022)
- [Photo-Sketching: Inferring Contour Drawings from Images](https://mlsanity.com/paper/1901.00542/) (2019)
- [Deep Learning for Free-Hand Sketch: A Survey](https://mlsanity.com/paper/2001.02600/) (2020)
- [Learning to Sketch with Shortcut Cycle Consistency](https://mlsanity.com/paper/1805.00247/) (2018)
- [Creative Sketch Generation](https://mlsanity.com/paper/2011.10039/) (2020)
- [Learning Deep Sketch Abstraction](https://mlsanity.com/paper/1804.04804/) (2018)
- [Why Do Line Drawings Work? A Realism Hypothesis](https://mlsanity.com/paper/2002.06260/) (2020)
- [Learning to Draw: Emergent Communication through Sketching](https://mlsanity.com/paper/2106.02067/) (2021)
- [SketchLattice: Latticed Representation for Sketch Manipulation](https://mlsanity.com/paper/2108.11636/) (2021)

## Built on by (most cited first, 30 of 64)

- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Models](https://mlsanity.com/paper/2311.16933/) (2023)
- [SVGDreamer: Text Guided SVG Generation with Diffusion Model](https://mlsanity.com/paper/2312.16476/) (2023)
- [Word-As-Image for Semantic Typography](https://mlsanity.com/paper/2303.01818/) (2023)
- [Concept Decomposition for Visual Exploration and Inspiration](https://mlsanity.com/paper/2305.18203/) (2023)
- [DiffSketcher: Text Guided Vector Sketch Synthesis through Latent Diffusion Models](https://mlsanity.com/paper/2306.14685/) (2023)
- [Breathing Life Into Sketches Using Text-to-Video Priors](https://mlsanity.com/paper/2311.13608/) (2023)
- [RT-Sketch: Goal-Conditioned Imitation Learning from Hand-Drawn Sketches](https://mlsanity.com/paper/2403.02709/) (2024)
- [SketchAgent: Language-Driven Sequential Sketch Generation](https://mlsanity.com/paper/2411.17673/) (2024)
- [ConvNet vs Transformer, Supervised vs CLIP: Beyond ImageNet Accuracy](https://mlsanity.com/paper/2311.09215/) (2023)
- [NeuralSVG: An Implicit Representation for Text-to-Vector Generation](https://mlsanity.com/paper/2501.03992/) (2025)
- [SwiftSketch: A Diffusion Model for Image-to-Vector Sketch Generation](https://mlsanity.com/paper/2502.08642/) (2025)
- [SketchINR: A First Look into Sketches as Implicit Neural Representations](https://mlsanity.com/paper/2403.09344/) (2024)
- [Render-in-the-Loop: Vector Graphics Generation via Visual Self-Feedback](https://mlsanity.com/paper/2604.20730/) (2026)
- [Diff3DS: Generating View-Consistent 3D Sketch via Differentiable Curve Rendering](https://mlsanity.com/paper/2405.15305/) (2024)
- [Multi-Object Sketch Animation with Grouping and Motion Trajectory Priors](https://mlsanity.com/paper/2508.15535/) (2025)
- [FlipSketch: Flipping Static Drawings to Text-Guided Sketch Animations](https://mlsanity.com/paper/2411.10818/) (2024)
- [Image Editing with Diffusion Models: A Survey](https://mlsanity.com/paper/2504.13226/) (2025)
- [VectorPainter: Advanced Stylized Vector Graphics Synthesis Using Stroke-Style Priors](https://mlsanity.com/paper/2405.02962/) (2024)
- [Reconstructive Latent-Space Neural Radiance Fields for Efficient 3D Scene Representations](https://mlsanity.com/paper/2310.17880/) (2023)
- [LineArt: A Knowledge-guided Training-free High-quality Appearance Transfer for Design Drawing with Diffusion Model](https://mlsanity.com/paper/2412.11519/) (2024)
- [ViewCraft3D: High-Fidelity and View-Consistent 3D Vector Graphics Synthesis](https://mlsanity.com/paper/2505.19492/) (2025)
- [Sketch Video Synthesis](https://mlsanity.com/paper/2311.15306/) (2023)
- [Open Vocabulary Semantic Scene Sketch Understanding](https://mlsanity.com/paper/2312.12463/) (2023)
- [WIR3D: Visually-Informed and Geometry-Aware 3D Shape Abstraction](https://mlsanity.com/paper/2505.04813/) (2025)
- [Neural Image Abstraction Using Long Smoothing B-Splines](https://mlsanity.com/paper/2511.05360/) (2025)
- [Fine-Tuned but Zero-Shot 3D Shape Sketch View Similarity and Retrieval](https://mlsanity.com/paper/2306.08541/) (2023)
- [Emergent Communication in Interactive Sketch Question Answering](https://mlsanity.com/paper/2310.15597/) (2023)
- [CraftSVG: Multi-Object Text-to-SVG Synthesis via Layout Guided Diffusion](https://mlsanity.com/paper/2404.00412/) (2024)
- [MixSA: Training-free Reference-based Sketch Extraction via Mixture-of-Self-Attention](https://mlsanity.com/paper/2501.00816/) (2025)

## Abstract (arXiv)

In this paper, we present a method for converting a given scene image into a sketch using different types and multiple levels of abstraction. We distinguish between two types of abstraction. The first considers the fidelity of the sketch, varying its representation from a more precise portrayal of the input to a looser depiction. The second is defined by the visual simplicity of the sketch, moving from a detailed depiction to a sparse sketch. Using an explicit disentanglement into two abstraction axes -- and multiple levels for each one -- provides users additional control over selecting the desired sketch based on their personal goals and preferences. To form a sketch at a given level of fidelity and simplification, we train two MLP networks. The first network learns the desired placement of strokes, while the second network learns to gradually remove strokes from the sketch without harming its recognizability and semantics. Our approach is able to generate sketches of complex scenes including those with complex backgrounds (e.g., natural and urban settings) and subjects (e.g., animals and people) while depicting gradual abstractions of the input scene in terms of fidelity and simplicity.