# AISFormer: Amodal Instance Segmentation with Transformer

arXiv: https://arxiv.org/abs/2210.06323 · HTML: https://mlsanity.com/paper/2210.06323/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 44 references that are in the map.
- First posted on arXiv 12 Oct 2022.

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Cascade R-CNN: Delving into High Quality Object Detection](https://mlsanity.com/paper/1712.00726/) (2017)
- [Designing Network Design Spaces](https://mlsanity.com/paper/2003.13678/) (2020)
- [Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation](https://mlsanity.com/paper/2012.07177/) (2020)
- [Hybrid Task Cascade for Instance Segmentation](https://mlsanity.com/paper/1901.07518/) (2019)
- [YOLACT: Real-time Instance Segmentation](https://mlsanity.com/paper/1904.02689/) (2019)
- [Dynamic Filter Networks](https://mlsanity.com/paper/1605.09673/) (2016)
- [Associative Embedding: End-to-End Learning for Joint Detection and Grouping](https://mlsanity.com/paper/1611.05424/) (2016)
- [CenterMask : Real-Time Anchor-Free Instance Segmentation](https://mlsanity.com/paper/1911.06667/) (2019)
- [PolarMask: Single Shot Instance Segmentation with Polar Representation](https://mlsanity.com/paper/1909.13226/) (2019)
- [Instances as Queries](https://mlsanity.com/paper/2105.01928/) (2021)
- [BlendMask: Top-Down Meets Bottom-Up for Instance Segmentation](https://mlsanity.com/paper/2001.00309/) (2020)
- [Fully Convolutional Instance-aware Semantic Segmentation](https://mlsanity.com/paper/1611.07709/) (2016)
- [Deep Occlusion-Aware Instance Segmentation with Overlapping BiLayers](https://mlsanity.com/paper/2103.12340/) (2021)
- [Semantic Amodal Segmentation](https://mlsanity.com/paper/1509.01329/) (2015)
- [TensorMask: A Foundation for Dense Object Segmentation](https://mlsanity.com/paper/1903.12174/) (2019)
- [Self-Supervised Scene De-occlusion](https://mlsanity.com/paper/2004.02788/) (2020)
- [Boundary-preserving Mask R-CNN](https://mlsanity.com/paper/2007.08921/) (2020)
- [SOLQ: Segmenting Objects by Learning Queries](https://mlsanity.com/paper/2106.02351/) (2021)
- [Mask Transfiner for High-Quality Instance Segmentation](https://mlsanity.com/paper/2111.13673/) (2021)
- [Amodal Instance Segmentation](https://mlsanity.com/paper/1604.08202/) (2016)
- [Learning to See the Invisible: End-to-End Trainable Amodal Instance Segmentation](https://mlsanity.com/paper/1804.08864/) (2018)
- [ISTR: End-to-End Instance Segmentation with Transformers](https://mlsanity.com/paper/2105.00637/) (2021)
- [RefineMask: Towards High-Quality Instance Segmentation with Fine-Grained Features](https://mlsanity.com/paper/2104.08569/) (2021)
- [Amodal Segmentation Based on Visible Region Segmentation and Shape Prior](https://mlsanity.com/paper/2012.05598/) (2020)

## Built on by (most cited first, 30 of 35)

- [Amodal Ground Truth and Completion in the Wild](https://mlsanity.com/paper/2312.17247/) (2023)
- [Coarse-to-Fine Amodal Segmentation with Shape Prior](https://mlsanity.com/paper/2308.16825/) (2023)
- [AerialFormer: Multi-resolution Transformer for Aerial Image Segmentation](https://mlsanity.com/paper/2306.06842/) (2023)
- [Using Diffusion Priors for Video Amodal Segmentation](https://mlsanity.com/paper/2412.04623/) (2024)
- [AmodalSynthDrive: A Synthetic Amodal Perception Dataset for Autonomous Driving](https://mlsanity.com/paper/2309.06547/) (2023)
- [ZEETAD: Adapting Pretrained Vision-Language Model for Zero-Shot End-to-End Temporal Action Detection](https://mlsanity.com/paper/2311.00729/) (2023)
- [Rethinking Amodal Video Segmentation from Learning Supervised Signals with Object-centric Representation](https://mlsanity.com/paper/2309.13248/) (2023)
- [Amodal Optical Flow](https://mlsanity.com/paper/2311.07761/) (2023)
- [TAO-Amodal: A Benchmark for Tracking Any Object Amodally](https://mlsanity.com/paper/2312.12433/) (2023)
- [Occlusion-Aware Seamless Segmentation](https://mlsanity.com/paper/2407.02182/) (2024)
- [ShapeFormer: Shape Prior Visible-to-Amodal Transformer-based Amodal Instance Segmentation](https://mlsanity.com/paper/2403.11376/) (2024)
- [Amodal Instance Segmentation with Diffusion Shape Prior Estimation](https://mlsanity.com/paper/2409.18256/) (2024)
- [PLUG: Revisiting Amodal Segmentation with Foundation Model and Hierarchical Focus](https://mlsanity.com/paper/2405.16094/) (2024)
- [Unlocking Constraints: Source-Free Occlusion-Aware Seamless Segmentation](https://mlsanity.com/paper/2506.21198/) (2025)
- [Referring Layer Decomposition](https://mlsanity.com/paper/2602.19358/) (2026)
- [Completing Visual Objects via Bridging Generation and Segmentation](https://mlsanity.com/paper/2310.00808/) (2023)
- [LAC-Net: Linear-Fusion Attention-Guided Convolutional Network for Accurate Robotic Grasping Under the Occlusion](https://mlsanity.com/paper/2408.03238/) (2024)
- [Amodal SAM: A Unified Amodal Segmentation Framework with Generalization](https://mlsanity.com/paper/2604.20748/) (2026)
- [SolarFormer: Multi-scale Transformer for Solar PV Profiling](https://mlsanity.com/paper/2310.20057/) (2023)
- [CarcassFormer: An End-to-end Transformer-based Framework for Simultaneous Localization, Segmentation and Classification of Poultry Carcass Defect](https://mlsanity.com/paper/2404.11429/) (2024)
- [Sequential Amodal Segmentation via Cumulative Occlusion Learning](https://mlsanity.com/paper/2405.05791/) (2024)
- [Hyper-Transformer for Amodal Completion](https://mlsanity.com/paper/2405.19949/) (2024)
- [Illustrator's Depth: Monocular Layer Index Prediction for Image Decomposition](https://mlsanity.com/paper/2511.17454/) (2025)
- [ViTA-Seg: Vision Transformer for Amodal Segmentation in Robotics](https://mlsanity.com/paper/2512.09510/) (2025)
- [Learnable Shape Prototypes with Occlusion-Geometry-Guided Injection for Amodal Instance Segmentation](https://mlsanity.com/paper/2605.24533/) (2026)
- [S3Former: Self-supervised High-resolution Transformer for Solar PV Profiling](https://mlsanity.com/paper/2405.04489/) (2024)
- [Mask Guided Gated Convolution for Amodal Content Completion](https://mlsanity.com/paper/2407.15203/) (2024)
- [Amodal Segmentation for Laparoscopic Surgery Video Instruments](https://mlsanity.com/paper/2408.01067/) (2024)
- [Track Anything Behind Everything: Zero-Shot Amodal Video Object Segmentation](https://mlsanity.com/paper/2411.19210/) (2024)
- [A2VIS: Amodal-Aware Approach to Video Instance Segmentation](https://mlsanity.com/paper/2412.01147/) (2024)

## Abstract (arXiv)

Amodal Instance Segmentation (AIS) aims to segment the region of both visible and possible occluded parts of an object instance. While Mask R-CNN-based AIS approaches have shown promising results, they are unable to model high-level features coherence due to the limited receptive field. The most recent transformer-based models show impressive performance on vision tasks, even better than Convolution Neural Networks (CNN). In this work, we present AISFormer, an AIS framework, with a Transformer-based mask head. AISFormer explicitly models the complex coherence between occluder, visible, amodal, and invisible masks within an object's regions of interest by treating them as learnable queries. Specifically, AISFormer contains four modules: (i) feature encoding: extract ROI and learn both short-range and long-range visual features. (ii) mask transformer decoding: generate the occluder, visible, and amodal mask query embeddings by a transformer decoder (iii) invisible mask embedding: model the coherence between the amodal and visible masks, and (iv) mask predicting: estimate output masks including occluder, visible, amodal and invisible. We conduct extensive experiments and ablation studies on three challenging benchmarks i.e. KINS, D2SA, and COCOA-cls to evaluate the effectiveness of AISFormer. The code is available at: https://github.com/UARK-AICV/AISFormer