# MixSA: Training-free Reference-based Sketch Extraction via Mixture-of-Self-Attention

arXiv: https://arxiv.org/abs/2501.00816 · HTML: https://mlsanity.com/paper/2501.00816/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 59 references that are in the map.
- First posted on arXiv 1 Jan 2025.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2308.06721/) (2023)
- [Demystifying MMD GANs](https://mlsanity.com/paper/1801.01401/) (2018)
- [Null-text Inversion for Editing Real Images using Guided Diffusion Models](https://mlsanity.com/paper/2211.09794/) (2022)
- [Holistically-Nested Edge Detection](https://mlsanity.com/paper/1504.06375/) (2015)
- [FateZero: Fusing Attentions for Zero-shot Text-based Video Editing](https://mlsanity.com/paper/2303.09535/) (2023)
- [ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generation](https://mlsanity.com/paper/2302.13848/) (2023)
- [U2-Net: Going Deeper with Nested U-Structure for Salient Object Detection](https://mlsanity.com/paper/2005.09007/) (2020)
- [Inversion-Based Style Transfer with Diffusion Models](https://mlsanity.com/paper/2211.13203/) (2022)
- [Style Aligned Image Generation via Shared Attention](https://mlsanity.com/paper/2312.02133/) (2023)
- [Swapping Autoencoder for Deep Image Manipulation](https://mlsanity.com/paper/2007.00653/) (2020)
- [FreeU: Free Lunch in Diffusion U-Net](https://mlsanity.com/paper/2309.11497/) (2023)
- [Style Injection in Diffusion: A Training-free Approach for Adapting Large-scale Diffusion Models for Style Transfer](https://mlsanity.com/paper/2312.09008/) (2023)
- [PolarMask: Single Shot Instance Segmentation with Polar Representation](https://mlsanity.com/paper/1909.13226/) (2019)
- [InstantStyle: Free Lunch towards Style-Preserving in Text-to-Image Generation](https://mlsanity.com/paper/2404.02733/) (2024)
- [StyleDiffusion: Controllable Disentangled Style Transfer via Diffusion Models](https://mlsanity.com/paper/2308.07863/) (2023)
- [Cross-Image Attention for Zero-Shot Appearance Transfer](https://mlsanity.com/paper/2311.03335/) (2023)
- [Dense Extreme Inception Network: Towards a Robust CNN Model for Edge Detection](https://mlsanity.com/paper/1909.01955/) (2019)
- [Learning to generate line drawings that convey geometry and semantics](https://mlsanity.com/paper/2203.12691/) (2022)
- [CLIPascene: Scene Sketching with Different Types and Levels of Abstraction](https://mlsanity.com/paper/2211.17256/) (2022)
- [Visual Style Prompting with Swapping Self-Attention](https://mlsanity.com/paper/2402.12974/) (2024)
- [Tiny and Efficient Model for the Edge Detection Generalization](https://mlsanity.com/paper/2308.06468/) (2023)
- [The Treasure Beneath Multiple Annotations: An Uncertainty-aware Edge Detector](https://mlsanity.com/paper/2303.11828/) (2023)
- [Adversarial Open Domain Adaptation for Sketch-to-Photo Synthesis](https://mlsanity.com/paper/2104.05703/) (2021)

## Built on by (most cited first, 3 of 3)

- [StrokeFusion: Vector Sketch Generation via Joint Stroke-UDF Encoding and Latent Sequence Diffusion](https://mlsanity.com/paper/2503.23752/) (2025)
- [Stroke2Sketch: Harnessing Stroke Attributes for Training-Free Sketch Generation](https://mlsanity.com/paper/2510.16319/) (2025)
- [Text to Sketch Generation with Multi-Styles](https://mlsanity.com/paper/2511.04123/) (2025)

## Abstract (arXiv)

Current sketch extraction methods either require extensive training or fail to capture a wide range of artistic styles, limiting their practical applicability and versatility. We introduce Mixture-of-Self-Attention (MixSA), a training-free sketch extraction method that leverages strong diffusion priors for enhanced sketch perception. At its core, MixSA employs a mixture-of-self-attention technique, which manipulates self-attention layers by substituting the keys and values with those from reference sketches. This allows for the seamless integration of brushstroke elements into initial outline images, offering precise control over texture density and enabling interpolation between styles to create novel, unseen styles. By aligning brushstroke styles with the texture and contours of colored images, particularly in late decoder layers handling local textures, MixSA addresses the common issue of color averaging by adjusting initial outlines. Evaluated with various perceptual metrics, MixSA demonstrates superior performance in sketch quality, flexibility, and applicability. This approach not only overcomes the limitations of existing methods but also empowers users to generate diverse, high-fidelity sketches that more accurately reflect a wide range of artistic expressions.