# InstaFormer: Instance-Aware Image-to-Image Translation with Transformer

arXiv: https://arxiv.org/abs/2203.16248 · HTML: https://mlsanity.com/paper/2203.16248/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 61 of its 71 references that are in the map.
- First posted on arXiv 30 Mar 2022.

## Builds on (most cited first, 30 of 61)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [On the Spectral Bias of Neural Networks](https://mlsanity.com/paper/1806.08734/) (2018)
- [Image Super-Resolution Using Deep Convolutional Networks](https://mlsanity.com/paper/1501.00092/) (2014)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [StarGAN v2: Diverse Image Synthesis for Multiple Domains](https://mlsanity.com/paper/1912.01865/) (2019)
- [CyCADA: Cycle-Consistent Adversarial Domain Adaptation](https://mlsanity.com/paper/1711.03213/) (2017)
- [StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation](https://mlsanity.com/paper/1711.09020/) (2017)
- [Accurate Image Super-Resolution Using Very Deep Convolutional Networks](https://mlsanity.com/paper/1511.04587/) (2015)
- [Pre-Trained Image Processing Transformer](https://mlsanity.com/paper/2012.00364/) (2020)

## Built on by (most cited first, 28 of 28)

- [Milestones in Autonomous Driving and Intelligent Vehicles Part I: Control, Computing System Design, Communication, HD Map, Testing, and Human Behaviors](https://mlsanity.com/paper/2305.11239/) (2023)
- [DiffV2IR: Visible-to-Infrared Diffusion Model via Vision-Language Understanding](https://mlsanity.com/paper/2503.19012/) (2025)
- [Transformer-based Generative Adversarial Networks in Computer Vision: A Comprehensive Survey](https://mlsanity.com/paper/2302.08641/) (2023)
- [C-DiffSET: Leveraging Latent Diffusion for SAR-to-EO Image Translation with Confidence-Guided Reliable Object Generation](https://mlsanity.com/paper/2411.10788/) (2024)
- [Anatomically constrained CT image translation for heterogeneous blood vessel segmentation](https://mlsanity.com/paper/2210.01713/) (2022)
- [One-for-All: Towards Universal Domain Translation with a Single StyleGAN](https://mlsanity.com/paper/2310.14222/) (2023)
- [Weight Copy and Low-Rank Adaptation for Few-Shot Distillation of Vision Transformers](https://mlsanity.com/paper/2404.09326/) (2024)
- [Transformer-based Image and Video Inpainting: Current Challenges and Future Directions](https://mlsanity.com/paper/2407.00226/) (2024)
- [Bridging Day and Night: Target-Class Hallucination Suppression in Unpaired Image Translation](https://mlsanity.com/paper/2602.15383/) (2026)
- [LANIT: Language-Driven Image-to-Image Translation for Unlabeled Data](https://mlsanity.com/paper/2208.14889/) (2022)
- [SHUNIT: Style Harmonization for Unpaired Image-to-Image Translation](https://mlsanity.com/paper/2301.04685/) (2023)
- [RADIO: Reference-Agnostic Dubbing Video Synthesis](https://mlsanity.com/paper/2309.01950/) (2023)
- [Masked Discriminators for Content-Consistent Unpaired Image-to-Image Translation](https://mlsanity.com/paper/2309.13188/) (2023)
- [Improving 2D-3D Dense Correspondences with Diffusion Models for 6D Object Pose Estimation](https://mlsanity.com/paper/2402.06436/) (2024)
- [Reliable Multi-modal Medical Image-to-image Translation Independent of Pixel-wise Aligned Data](https://mlsanity.com/paper/2408.14270/) (2024)
- [CSHNet: A Novel Information Asymmetric Image Translation Method](https://mlsanity.com/paper/2501.10197/) (2025)
- [DSI2I: Dense Style for Unpaired Image-to-Image Translation](https://mlsanity.com/paper/2212.13253/) (2022)
- [AptSim2Real: Approximately-Paired Sim-to-Real Image Translation](https://mlsanity.com/paper/2303.12704/) (2023)
- [Panoramic Image-to-Image Translation](https://mlsanity.com/paper/2304.04960/) (2023)
- [Improving Object Detection via Local-global Contrastive Learning](https://mlsanity.com/paper/2410.05058/) (2024)
- [LatticeVision: Image to Image Networks for Modeling Non-Stationary Spatial Data](https://mlsanity.com/paper/2505.09803/) (2025)
- [Domain Agnostic Image-to-image Translation using Low-Resolution Conditioning](https://mlsanity.com/paper/2305.05023/) (2023)
- [Enhancing Visual Domain Adaptation with Source Preparation](https://mlsanity.com/paper/2306.10142/) (2023)
- [Learning Nuclei Representations with Masked Image Modelling](https://mlsanity.com/paper/2306.17116/) (2023)
- [Disentangled Contrastive Image Translation for Nighttime Surveillance](https://mlsanity.com/paper/2307.05038/) (2023)
- [Hierarchical Conditional Semi-Paired Image-to-Image Translation For Multi-Task Image Defect Correction On Shopping Websites](https://mlsanity.com/paper/2309.05883/) (2023)
- [DT-NeRF: A Diffusion and Transformer-Based Optimization Approach for Neural Radiance Fields in 3D Reconstruction](https://mlsanity.com/paper/2509.17232/) (2025)
- [BrainPuzzle: Hybrid Physics and Data-Driven Reconstruction for Transcranial Ultrasound Tomography](https://mlsanity.com/paper/2510.20029/) (2025)

## Abstract (arXiv)

We present a novel Transformer-based network architecture for instance-aware image-to-image translation, dubbed InstaFormer, to effectively integrate global- and instance-level information. By considering extracted content features from an image as tokens, our networks discover global consensus of content features by considering context information through a self-attention module in Transformers. By augmenting such tokens with an instance-level feature extracted from the content feature with respect to bounding box information, our framework is capable of learning an interaction between object instances and the global image, thus boosting the instance-awareness. We replace layer normalization (LayerNorm) in standard Transformers with adaptive instance normalization (AdaIN) to enable a multi-modal translation with style codes. In addition, to improve the instance-awareness and translation quality at object regions, we present an instance-level content contrastive loss defined between input and translated image. We conduct experiments to demonstrate the effectiveness of our InstaFormer over the latest methods and provide extensive ablation studies.