# Eliminating Gradient Conflict in Reference-based Line-Art Colorization

arXiv: https://arxiv.org/abs/2207.06095 · HTML: https://mlsanity.com/paper/2207.06095/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 61 references that are in the map.
- First posted on arXiv 13 Jul 2022.

## Builds on (most cited first, 30 of 51)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [Point Transformer](https://mlsanity.com/paper/2011.00931/) (2020)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Semantic Image Synthesis with Spatially-Adaptive Normalization](https://mlsanity.com/paper/1903.07291/) (2019)
- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [StarGAN v2: Diverse Image Synthesis for Multiple Domains](https://mlsanity.com/paper/1912.01865/) (2019)
- [Least Squares Generative Adversarial Networks](https://mlsanity.com/paper/1611.04076/) (2016)
- [CvT: Introducing Convolutions to Vision Transformers](https://mlsanity.com/paper/2103.15808/) (2021)
- [Self-Attention Generative Adversarial Networks](https://mlsanity.com/paper/1805.08318/) (2018)
- [PCT: Point cloud transformer](https://mlsanity.com/paper/2012.09688/) (2020)
- [Efficient Content-Based Sparse Attention with Routing Transformers](https://mlsanity.com/paper/2003.05997/) (2020)

## Built on by (most cited first, 25 of 25)

- [RenAIssance: A Survey into AI Text-to-Image Generation in the Era of Large Model](https://mlsanity.com/paper/2309.00810/) (2023)
- [AniDoc: Animation Creation Made Easier](https://mlsanity.com/paper/2412.14173/) (2024)
- [MangaNinja: Line Art Colorization with Precise Reference Following](https://mlsanity.com/paper/2501.08332/) (2025)
- [Transformer-based Generative Adversarial Networks in Computer Vision: A Comprehensive Survey](https://mlsanity.com/paper/2302.08641/) (2023)
- [Follow-Your-Color: Multi-Instance Sketch Colorization](https://mlsanity.com/paper/2503.16948/) (2025)
- [AnimeDiffusion: Anime Face Line Drawing Colorization via Diffusion Models](https://mlsanity.com/paper/2303.11137/) (2023)
- [GenView: Enhancing View Quality with Pretrained Generative Model for Self-Supervised Learning](https://mlsanity.com/paper/2403.12003/) (2024)
- [ColorizeDiffusion: Adjustable Sketch Colorization with Reference Image and Text](https://mlsanity.com/paper/2401.01456/) (2024)
- [Automatic Controllable Colorization via Imagination](https://mlsanity.com/paper/2404.05661/) (2024)
- [ColorFlow: Retrieval-Augmented Image Sequence Colorization](https://mlsanity.com/paper/2412.11815/) (2024)
- [Attention-Aware Anime Line Drawing Colorization](https://mlsanity.com/paper/2212.10988/) (2022)
- [Image Referenced Sketch Colorization Based on Animation Creation Workflow](https://mlsanity.com/paper/2502.19937/) (2025)
- [Thin-Plate Spline-based Interpolation for Animation Line Inbetweening](https://mlsanity.com/paper/2408.09131/) (2024)
- [From Fragment to One Piece: A Survey on AI-Driven Graphic Design](https://mlsanity.com/paper/2503.18641/) (2025)
- [DACoN: DINO for Anime Paint Bucket Colorization with Any Number of Reference Images](https://mlsanity.com/paper/2509.14685/) (2025)
- [SketchColour: Channel Concat Guided DiT-based Sketch-to-Colour Pipeline for 2D Animation](https://mlsanity.com/paper/2507.01586/) (2025)
- [OmniColor: A Unified Framework for Multi-modal Lineart Colorization](https://mlsanity.com/paper/2603.27531/) (2026)
- [UMAIR-FPS: User-aware Multi-modal Animation Illustration Recommendation Fusion with Painting Style](https://mlsanity.com/paper/2402.10381/) (2024)
- [Computer-aided Colorization State-of-the-science: A Survey](https://mlsanity.com/paper/2410.02288/) (2024)
- [ColorizeDiffusion v2: Enhancing Reference-based Sketch Colorization Through Separating Utilities](https://mlsanity.com/paper/2504.06895/) (2025)
- [Few-shot Semantic Encoding and Decoding for Video Surveillance](https://mlsanity.com/paper/2505.07381/) (2025)
- [SSIMBaD: Sigma Scaling with SSIM-Guided Balanced Diffusion for AnimeFace Colorization](https://mlsanity.com/paper/2506.04283/) (2025)
- [One-shot Embroidery Customization via Contrastive LoRA Modulation](https://mlsanity.com/paper/2509.18948/) (2025)
- [Uni-Animator: Towards Unified Visual Colorization](https://mlsanity.com/paper/2602.23191/) (2026)
- [Towards High-resolution and Disentangled Reference-based Sketch Colorization](https://mlsanity.com/paper/2603.05971/) (2026)

## Abstract (arXiv)

Reference-based line-art colorization is a challenging task in computer vision. The color, texture, and shading are rendered based on an abstract sketch, which heavily relies on the precise long-range dependency modeling between the sketch and reference. Popular techniques to bridge the cross-modal information and model the long-range dependency employ the attention mechanism. However, in the context of reference-based line-art colorization, several techniques would intensify the existing training difficulty of attention, for instance, self-supervised training protocol and GAN-based losses. To understand the instability in training, we detect the gradient flow of attention and observe gradient conflict among attention branches. This phenomenon motivates us to alleviate the gradient issue by preserving the dominant gradient branch while removing the conflict ones. We propose a novel attention mechanism using this training strategy, Stop-Gradient Attention (SGA), outperforming the attention baseline by a large margin with better training stability. Compared with state-of-the-art modules in line-art colorization, our approach demonstrates significant improvements in Fr'echet Inception Distance (FID, up to 27.21%) and structural similarity index measure (SSIM, up to 25.67%) on several benchmarks. The code of SGA is available at https://github.com/kunkun0w0/SGA .