# S3POT: Contrast-Driven Face Occlusion Segmentation via Self-Supervised Prompt Learning

arXiv: https://arxiv.org/abs/2602.00635 · HTML: https://mlsanity.com/paper/2602.00635/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 31 references that are in the map.
- First posted on arXiv 31 Jan 2026.

## Builds on (most cited first, 23 of 23)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [LISA: Reasoning Segmentation via Large Language Model](https://mlsanity.com/paper/2308.00692/) (2023)
- [Encoding in Style: a StyleGAN Encoder for Image-to-Image Translation](https://mlsanity.com/paper/2008.00951/) (2020)
- [MaskGAN: Towards Diverse and Interactive Facial Image Manipulation](https://mlsanity.com/paper/1907.11922/) (2019)
- [Personalize Segment Anything Model with One Shot](https://mlsanity.com/paper/2305.03048/) (2023)
- [Matcher: Segment Anything with One Shot Using All-Purpose Feature Matching](https://mlsanity.com/paper/2305.13310/) (2023)
- [Open-Vocabulary SAM: Segment and Recognize Twenty-thousand Classes Interactively](https://mlsanity.com/paper/2401.02955/) (2024)
- [VRP-SAM: SAM with Visual Reference Prompt](https://mlsanity.com/paper/2402.17726/) (2024)
- [Fine-Grained Face Swapping via Regional GAN Inversion](https://mlsanity.com/paper/2211.14068/) (2022)
- [Deep Human Parsing with Active Template Regression](https://mlsanity.com/paper/1503.02391/) (2015)
- [See, Say, and Segment: Teaching LMMs to Overcome False Premises](https://mlsanity.com/paper/2312.08366/) (2023)
- [Realistic and Efficient Face Swapping: A Unified Approach with Diffusion Models](https://mlsanity.com/paper/2409.07269/) (2024)
- [Bridge the Points: Graph-based Few-shot Segment Anything Semantically](https://mlsanity.com/paper/2410.06964/) (2024)
- [Face Parsing with RoI Tanh-Warping](https://mlsanity.com/paper/1906.01342/) (2019)
- [Decoupled Multi-task Learning with Cyclical Self-Regulation for Face Parsing](https://mlsanity.com/paper/2203.14448/) (2022)
- [SegFace: Face Segmentation of Long-Tail Classes](https://mlsanity.com/paper/2412.08647/) (2024)
- [Facial Attribute Transformers for Precise and Robust Makeup Transfer](https://mlsanity.com/paper/2104.02894/) (2021)
- [Interlinked Convolutional Neural Networks for Face Parsing](https://mlsanity.com/paper/1806.02479/) (2018)
- [Face Parsing via Recurrent Propagation](https://mlsanity.com/paper/1708.01936/) (2017)
- [Learning Disentangled Representation for One-shot Progressive Face Swapping](https://mlsanity.com/paper/2203.12985/) (2022)
- [Delving into High-Quality Synthetic Face Occlusion Segmentation Datasets](https://mlsanity.com/paper/2205.06218/) (2022)

## Abstract (arXiv)

Existing face parsing methods usually misclassify occlusions as facial components. This is because occlusion is a high-level concept, it does not refer to a concrete category of object. Thus, constructing a real-world face dataset covering all categories of occlusion object is almost impossible and accurate mask annotation is labor-intensive. To deal with the problems, we present S3POT, a contrast-driven framework synergizing face generation with self-supervised spatial prompting, to achieve occlusion segmentation. The framework is inspired by the insights: 1) Modern face generators' ability to realistically reconstruct occluded regions, creating an image that preserve facial geometry while eliminating occlusion, and 2) Foundation segmentation models' (e.g., SAM) capacity to extract precise mask when provided with appropriate prompts. In particular, S3POT consists of three modules: Reference Generation (RF), Feature enhancement (FE), and Prompt Selection (PS). First, a reference image is produced by RF using structural guidance from parsed mask. Second, FE performs contrast of tokens between raw and reference images to obtain an initial prompt, then modifies image features with the prompt by cross-attention. Third, based on the enhanced features, PS constructs a set of positive and negative prompts and screens them with a self-attention network for a mask decoder. The network is learned under the guidance of three novel and complementary objective functions without occlusion ground truth mask involved. Extensive experiments on a dedicatedly collected dataset demonstrate S3POT's superior performance and the effectiveness of each module.