# DynaMITe: Dynamic Query Bootstrapping for Multi-object Interactive Segmentation Transformer

arXiv: https://arxiv.org/abs/2304.06668 · HTML: https://mlsanity.com/paper/2304.06668/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 48 references that are in the map.
- First posted on arXiv 13 Apr 2023.

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [FCOS: Fully Convolutional One-Stage Object Detection](https://mlsanity.com/paper/1904.01355/) (2019)
- [Deep High-Resolution Representation Learning for Visual Recognition](https://mlsanity.com/paper/1908.07919/) (2019)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [ScribbleSup: Scribble-Supervised Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1604.05144/) (2016)
- [Conditional Convolutions for Instance Segmentation](https://mlsanity.com/paper/2003.05664/) (2020)
- [Associative Embedding: End-to-End Learning for Joint Detection and Grouping](https://mlsanity.com/paper/1611.05424/) (2016)
- [Local Relation Networks for Image Recognition](https://mlsanity.com/paper/1904.11491/) (2019)
- [Reviving Iterative Training with Mask Guidance for Interactive Segmentation](https://mlsanity.com/paper/2102.06583/) (2021)
- [Deep Interactive Object Selection](https://mlsanity.com/paper/1603.04042/) (2016)
- [f-BRS: Rethinking Backpropagating Refinement for Interactive Segmentation](https://mlsanity.com/paper/2001.10331/) (2020)
- [FocalClick: Towards Practical Interactive Image Segmentation](https://mlsanity.com/paper/2204.02574/) (2022)
- [Instance Segmentation by Jointly Optimizing Spatial Embeddings and Clustering Bandwidth](https://mlsanity.com/paper/1906.11109/) (2019)
- [Annotating Object Instances with a Polygon-RNN](https://mlsanity.com/paper/1704.05548/) (2017)
- [Recurrent Pixel Embedding for Instance Grouping](https://mlsanity.com/paper/1712.08273/) (2017)
- [Iteratively Trained Interactive Segmentation](https://mlsanity.com/paper/1805.04398/) (2018)
- [PseudoClick: Interactive Image Segmentation with Click Imitation](https://mlsanity.com/paper/2207.05282/) (2022)
- [EdgeFlow: Achieving Practical Interactive Segmentation with Edge-Guided Flow](https://mlsanity.com/paper/2109.09406/) (2021)

## Built on by (most cited first, 14 of 14)

- [AGILE3D: Attention Guided Interactive Multi-object 3D Segmentation](https://mlsanity.com/paper/2306.00977/) (2023)
- [AdaptiveClick: Clicks-aware Transformer with Adaptive Focal Loss for Interactive Image Segmentation](https://mlsanity.com/paper/2305.04276/) (2023)
- [Point-VOS: Pointing Up Video Object Segmentation](https://mlsanity.com/paper/2402.05917/) (2024)
- [MFP: Making Full Use of Probability Maps for Interactive Image Segmentation](https://mlsanity.com/paper/2404.18448/) (2024)
- [Interactive4D: Interactive 4D LiDAR Segmentation](https://mlsanity.com/paper/2410.08206/) (2024)
- [Refining Segmentation On-the-Fly: An Interactive Framework for Point Cloud Semantic Segmentation](https://mlsanity.com/paper/2403.06401/) (2024)
- [PVPUFormer: Probabilistic Visual Prompt Unified Transformer for Interactive Image Segmentation](https://mlsanity.com/paper/2306.06656/) (2023)
- [Mixed-Query Transformer: A Unified Image Segmentation Architecture](https://mlsanity.com/paper/2404.04469/) (2024)
- [Learning from Exemplars for Interactive Image Segmentation](https://mlsanity.com/paper/2406.11472/) (2024)
- [Order-aware Interactive Segmentation](https://mlsanity.com/paper/2410.12214/) (2024)
- [M2N2V2: Multi-Modal Unsupervised and Training-free Interactive Segmentation](https://mlsanity.com/paper/2503.16254/) (2025)
- [MMMS: Multi-Modal Multi-Surface Interactive Segmentation](https://mlsanity.com/paper/2509.12963/) (2025)
- [MOIS-SAM2: Exemplar-based Segment Anything Model 2 for multilesion interactive segmentation of neurofibromas in whole-body MRI](https://mlsanity.com/paper/2509.19277/) (2025)
- [DropClick: Semi-Automated One-Click Segmentation for Agricultural Robotic Data](https://mlsanity.com/paper/2609.03680/) (2026)

## Abstract (arXiv)

Most state-of-the-art instance segmentation methods rely on large amounts of pixel-precise ground-truth annotations for training, which are expensive to create. Interactive segmentation networks help generate such annotations based on an image and the corresponding user interactions such as clicks. Existing methods for this task can only process a single instance at a time and each user interaction requires a full forward pass through the entire deep network. We introduce a more efficient approach, called DynaMITe, in which we represent user interactions as spatio-temporal queries to a Transformer decoder with a potential to segment multiple object instances in a single iteration. Our architecture also alleviates any need to re-compute image features during refinement, and requires fewer interactions for segmenting multiple instances in a single image when compared to other methods. DynaMITe achieves state-of-the-art results on multiple existing interactive segmentation benchmarks, and also on the new multi-instance benchmark that we propose in this paper.