# AdaptiveClick: Clicks-aware Transformer with Adaptive Focal Loss for Interactive Image Segmentation

arXiv: https://arxiv.org/abs/2305.04276 · HTML: https://mlsanity.com/paper/2305.04276/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 88 references that are in the map.
- First posted on arXiv 7 May 2023.

## Builds on (most cited first, 30 of 41)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [Exploring Plain Vision Transformer Backbones for Object Detection](https://mlsanity.com/paper/2203.16527/) (2022)
- [The Medical Segmentation Decathlon](https://mlsanity.com/paper/2106.05735/) (2021)
- [Holistically-Nested Edge Detection](https://mlsanity.com/paper/1504.06375/) (2015)
- [Segment Everything Everywhere All at Once](https://mlsanity.com/paper/2304.06718/) (2023)
- [The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classification](https://mlsanity.com/paper/2107.02314/) (2021)
- [A survey of loss functions for semantic segmentation](https://mlsanity.com/paper/2006.14822/) (2020)
- [Reviving Iterative Training with Mask Guidance for Interactive Segmentation](https://mlsanity.com/paper/2102.06583/) (2021)
- [Deep Interactive Object Selection](https://mlsanity.com/paper/1603.04042/) (2016)
- [End-to-End Referring Video Object Segmentation with Multimodal Transformers](https://mlsanity.com/paper/2111.14821/) (2021)
- [Deep Extreme Cut: From Extreme Points to Object Segmentation](https://mlsanity.com/paper/1711.09081/) (2017)
- [f-BRS: Rethinking Backpropagating Refinement for Interactive Segmentation](https://mlsanity.com/paper/2001.10331/) (2020)
- [SimpleClick: Interactive Image Segmentation with Simple Vision Transformers](https://mlsanity.com/paper/2210.11006/) (2022)
- [DeepIGeoS: A Deep Interactive Geodesic Framework for Medical Image Segmentation](https://mlsanity.com/paper/1707.00652/) (2017)
- [Large-scale interactive object segmentation with human annotators](https://mlsanity.com/paper/1903.10830/) (2019)
- [FocalClick: Towards Practical Interactive Image Segmentation](https://mlsanity.com/paper/2204.02574/) (2022)
- [AdaptIS: Adaptive Instance Selection Network](https://mlsanity.com/paper/1909.07829/) (2019)
- [PolyLoss: A Polynomial Expansion Perspective of Classification Loss Functions](https://mlsanity.com/paper/2204.12511/) (2022)
- [Equalized Focal Loss for Dense Long-Tailed Object Detection](https://mlsanity.com/paper/2201.02593/) (2022)
- [PseudoClick: Interactive Image Segmentation with Click Imitation](https://mlsanity.com/paper/2207.05282/) (2022)

## Built on by (most cited first, 11 of 11)

- [RClicks: Realistic Click Simulation for Benchmarking Interactive Segmentation](https://mlsanity.com/paper/2410.11722/) (2024)
- [Refining Segmentation On-the-Fly: An Interactive Framework for Point Cloud Semantic Segmentation](https://mlsanity.com/paper/2403.06401/) (2024)
- [PVPUFormer: Probabilistic Visual Prompt Unified Transformer for Interactive Image Segmentation](https://mlsanity.com/paper/2306.06656/) (2023)
- [AI-SAM: Automatic and Interactive Segment Anything Model](https://mlsanity.com/paper/2312.03119/) (2023)
- [Structured Click Control in Transformer-based Interactive Segmentation](https://mlsanity.com/paper/2405.04009/) (2024)
- [SAM-REF: Introducing Image-Prompt Synergy during Interaction for Detail Enhancement in the Segment Anything Model](https://mlsanity.com/paper/2408.11535/) (2024)
- [Interactive Occlusion Boundary Estimation through Exploitation of Synthetic Data](https://mlsanity.com/paper/2408.15038/) (2024)
- [RP-SAM2: Refining Point Prompts for Stable Surgical Instrument Segmentation](https://mlsanity.com/paper/2504.07117/) (2025)
- [3D Fourier-based Global Feature Extraction for Hyperspectral Image Classification](https://mlsanity.com/paper/2603.16426/) (2026)
- [ClickAttention: Click Region Similarity Guided Interactive Segmentation](https://mlsanity.com/paper/2408.06021/) (2024)
- [VFM-ISRefiner: Towards Better Adapting Vision Foundation Models for Interactive Segmentation of Remote Sensing Images](https://mlsanity.com/paper/2512.00718/) (2025)

## Abstract (arXiv)

Interactive Image Segmentation (IIS) has emerged as a promising technique for decreasing annotation time. Substantial progress has been made in pre- and post-processing for IIS, but the critical issue of interaction ambiguity, notably hindering segmentation quality, has been under-researched. To address this, we introduce AdaptiveClick -- a click-aware transformer incorporating an adaptive focal loss that tackles annotation inconsistencies with tools for mask- and pixel-level ambiguity resolution. To the best of our knowledge, AdaptiveClick is the first transformer-based, mask-adaptive segmentation framework for IIS. The key ingredient of our method is the Click-Aware Mask-adaptive transformer Decoder (CAMD), which enhances the interaction between click and image features. Additionally, AdaptiveClick enables pixel-adaptive differentiation of hard and easy samples in the decision space, independent of their varying distributions. This is primarily achieved by optimizing a generalized Adaptive Focal Loss (AFL) with a theoretical guarantee, where two adaptive coefficients control the ratio of gradient values for hard and easy pixels. Our analysis reveals that the commonly used Focal and BCE losses can be considered special cases of the proposed AFL. With a plain ViT backbone, extensive experimental results on nine datasets demonstrate the superiority of AdaptiveClick compared to state-of-the-art methods. The source code is publicly available at https://github.com/lab206/AdaptiveClick.