# The Animation Transformer: Visual Correspondence via Segment Matching

arXiv: https://arxiv.org/abs/2109.02614 · HTML: https://mlsanity.com/paper/2109.02614/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 35 references that are in the map.
- First posted on arXiv 6 Sep 2021.

## Builds on (most cited first, 18 of 18)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [SuperGlue: Learning Feature Matching with Graph Neural Networks](https://mlsanity.com/paper/1911.11763/) (2019)
- [Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks](https://mlsanity.com/paper/1703.10593/) (2017)
- [A Neural Representation of Sketch Drawings](https://mlsanity.com/paper/1704.03477/) (2017)
- [Space-Time Correspondence as a Contrastive Random Walk](https://mlsanity.com/paper/2006.14613/) (2020)
- [Tracking Emerges by Colorizing Videos](https://mlsanity.com/paper/1806.09594/) (2018)
- [Im2Vec: Synthesizing Vector Graphics without Vector Supervision](https://mlsanity.com/paper/2102.02798/) (2021)
- [ContextDesc: Local Descriptor Augmentation with Cross-Modality Context](https://mlsanity.com/paper/1904.04084/) (2019)
- [Deep Exemplar-based Video Colorization](https://mlsanity.com/paper/1906.09909/) (2019)
- [Creative Sketch Generation](https://mlsanity.com/paper/2011.10039/) (2020)
- [Correspondence Networks with Adaptive Neighbourhood Consensus](https://mlsanity.com/paper/2003.12059/) (2020)
- [User-Guided Deep Anime Line Art Colorization with Conditional Adversarial Networks](https://mlsanity.com/paper/1808.03240/) (2018)
- [Deep Video Color Propagation](https://mlsanity.com/paper/1808.03232/) (2018)
- [Learning to Shadow Hand-drawn Sketches](https://mlsanity.com/paper/2002.11812/) (2020)
- [SketchDesc: Learning Local Sketch Descriptors for Multi-view Correspondence](https://mlsanity.com/paper/2001.05744/) (2020)
- [SketchZooms: Deep multi-view descriptors for matching line drawings](https://mlsanity.com/paper/1912.05019/) (2019)

## Built on by (most cited first, 20 of 20)

- [Eliminating Gradient Conflict in Reference-based Line-Art Colorization](https://mlsanity.com/paper/2207.06095/) (2022)
- [Zero-Shot Everything Sketch-Based Image Retrieval, and in Explainable Style](https://mlsanity.com/paper/2303.14348/) (2023)
- [Improving the Perceptual Quality of 2D Animation Interpolation](https://mlsanity.com/paper/2111.12792/) (2021)
- [Sakuga-42M Dataset: Scaling Up Cartoon Research](https://mlsanity.com/paper/2405.07425/) (2024)
- [Generative AI for Cel-Animation: A Survey](https://mlsanity.com/paper/2501.06250/) (2025)
- [MangaNinja: Line Art Colorization with Precise Reference Following](https://mlsanity.com/paper/2501.08332/) (2025)
- [Learning Inclusion Matching for Animation Paint Bucket Colorization](https://mlsanity.com/paper/2403.18342/) (2024)
- [AnimeRun: 2D Animation Visual Correspondence from Open Source 3D Movies](https://mlsanity.com/paper/2211.05709/) (2022)
- [Follow-Your-Color: Multi-Instance Sketch Colorization](https://mlsanity.com/paper/2503.16948/) (2025)
- [Deep Geometrized Cartoon Line Inbetweening](https://mlsanity.com/paper/2309.16643/) (2023)
- [Paint Bucket Colorization Using Anime Character Color Design Sheets](https://mlsanity.com/paper/2410.19424/) (2024)
- [Unsupervised Contour Tracking of Live Cells by Mechanical and Cycle Consistency Losses](https://mlsanity.com/paper/2303.08364/) (2023)
- [LinkTo-Anime: A 2D Animation Optical Flow Dataset from 3D Model Rendering](https://mlsanity.com/paper/2506.02733/) (2025)
- [DACoN: DINO for Anime Paint Bucket Colorization with Any Number of Reference Images](https://mlsanity.com/paper/2509.14685/) (2025)
- [OmniColor: A Unified Framework for Multi-modal Lineart Colorization](https://mlsanity.com/paper/2603.27531/) (2026)
- [LatentColorization: Latent Diffusion-Based Speaker Video Colorization](https://mlsanity.com/paper/2405.05707/) (2024)
- [Transformer-based Neuro-Animator for Qualitative Simulation of Soft Body Movement](https://mlsanity.com/paper/2408.15258/) (2024)
- [Computer-aided Colorization State-of-the-science: A Survey](https://mlsanity.com/paper/2410.02288/) (2024)
- [Region-Wise Correspondence Prediction between Manga Line Art Images](https://mlsanity.com/paper/2509.09501/) (2025)
- [PeCA: Palette Context Assisted Inference for Test-Time Paint-Bucket Colourisation on Animation Videos](https://mlsanity.com/paper/2608.00903/) (2026)

## Abstract (arXiv)

Visual correspondence is a fundamental building block on the way to building assistive tools for hand-drawn animation. However, while a large body of work has focused on learning visual correspondences at the pixel-level, few approaches have emerged to learn correspondence at the level of line enclosures (segments) that naturally occur in hand-drawn animation. Exploiting this structure in animation has numerous benefits: it avoids the intractable memory complexity of attending to individual pixels in high resolution images and enables the use of real-world animation datasets that contain correspondence information at the level of per-segment colors. To that end, we propose the Animation Transformer (AnT) which uses a transformer-based architecture to learn the spatial and visual relationships between segments across a sequence of images. AnT enables practical ML-assisted colorization for professional animation workflows and is publicly accessible as a creative tool in Cadmium.