# PVPUFormer: Probabilistic Visual Prompt Unified Transformer for Interactive Image Segmentation

arXiv: https://arxiv.org/abs/2306.06656 · HTML: https://mlsanity.com/paper/2306.06656/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 65 references that are in the map.
- First posted on arXiv 11 Jun 2023.

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1802.02611/) (2018)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [Segment Everything Everywhere All at Once](https://mlsanity.com/paper/2304.06718/) (2023)
- [The RSNA-ASNR-MICCAI BraTS 2021 Benchmark on Brain Tumor Segmentation and Radiogenomic Classification](https://mlsanity.com/paper/2107.02314/) (2021)
- [High-Resolution Representations for Labeling Pixels and Regions](https://mlsanity.com/paper/1904.04514/) (2019)
- [Reviving Iterative Training with Mask Guidance for Interactive Segmentation](https://mlsanity.com/paper/2102.06583/) (2021)
- [Deep Interactive Object Selection](https://mlsanity.com/paper/1603.04042/) (2016)
- [f-BRS: Rethinking Backpropagating Refinement for Interactive Segmentation](https://mlsanity.com/paper/2001.10331/) (2020)
- [SimpleClick: Interactive Image Segmentation with Simple Vision Transformers](https://mlsanity.com/paper/2210.11006/) (2022)
- [Large-scale interactive object segmentation with human annotators](https://mlsanity.com/paper/1903.10830/) (2019)
- [FocalClick: Towards Practical Interactive Image Segmentation](https://mlsanity.com/paper/2204.02574/) (2022)
- [Look Closer to Segment Better: Boundary Patch Refinement for Instance Segmentation](https://mlsanity.com/paper/2104.05239/) (2021)
- [PseudoClick: Interactive Image Segmentation with Click Imitation](https://mlsanity.com/paper/2207.05282/) (2022)
- [InterFormer: Real-time Interactive Image Segmentation](https://mlsanity.com/paper/2304.02942/) (2023)
- [Getting to 99% Accuracy in Interactive Segmentation](https://mlsanity.com/paper/2003.07932/) (2020)
- [CFR-ICL: Cascade-Forward Refinement with Iterative Click Loss for Interactive Image Segmentation](https://mlsanity.com/paper/2303.05620/) (2023)
- [Focused and Collaborative Feedback Integration for Interactive Image Segmentation](https://mlsanity.com/paper/2303.11880/) (2023)
- [Probabilistic Prompt Learning for Dense Prediction](https://mlsanity.com/paper/2304.00779/) (2023)
- [Active Pointly-Supervised Instance Segmentation](https://mlsanity.com/paper/2207.11493/) (2022)
- [Personal Fixations-Based Object Segmentation with Object Localization and Boundary Preservation](https://mlsanity.com/paper/2101.09014/) (2021)
- [Interactive Segmentation as Gaussian Process Classification](https://mlsanity.com/paper/2302.14578/) (2023)
- [DynaMITe: Dynamic Query Bootstrapping for Multi-object Interactive Segmentation Transformer](https://mlsanity.com/paper/2304.06668/) (2023)

## Built on by (most cited first, 2 of 2)

- [Cross-modal Context-aware Learning for Visual Prompt Guided Multimodal Image Understanding in Remote Sensing](https://mlsanity.com/paper/2512.11680/) (2025)
- [Scene Graph-guided SegCaptioning Transformer with Fine-grained Alignment for Controllable Video Segmentation and Captioning](https://mlsanity.com/paper/2603.20887/) (2026)

## Abstract (arXiv)

Integration of diverse visual prompts like clicks, scribbles, and boxes in interactive image segmentation significantly facilitates users' interaction as well as improves interaction efficiency. However, existing studies primarily encode the position or pixel regions of prompts without considering the contextual areas around them, resulting in insufficient prompt feedback, which is not conducive to performance acceleration. To tackle this problem, this paper proposes a simple yet effective Probabilistic Visual Prompt Unified Transformer (PVPUFormer) for interactive image segmentation, which allows users to flexibly input diverse visual prompts with the probabilistic prompt encoding and feature post-processing to excavate sufficient and robust prompt features for performance boosting. Specifically, we first propose a Probabilistic Prompt-unified Encoder (PPuE) to generate a unified one-dimensional vector by exploring both prompt and non-prompt contextual information, offering richer feedback cues to accelerate performance improvement. On this basis, we further present a Prompt-to-Pixel Contrastive (P2C) loss to accurately align both prompt and pixel features, bridging the representation gap between them to offer consistent feature representations for mask prediction. Moreover, our approach designs a Dual-cross Merging Attention (DMA) module to implement bidirectional feature interaction between image and prompt features, generating notable features for performance improvement. A comprehensive variety of experiments on several challenging datasets demonstrates that the proposed components achieve consistent improvements, yielding state-of-the-art interactive segmentation performance. Our code is available at https://github.com/XuZhang1211/PVPUFormer.