# Referring Image Matting

arXiv: https://arxiv.org/abs/2206.05149 · HTML: https://mlsanity.com/paper/2206.05149/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 81 references that are in the map.
- First posted on arXiv 10 Jun 2022.

## Builds on (most cited first, 30 of 48)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Modeling Context in Referring Expressions](https://mlsanity.com/paper/1608.00272/) (2016)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [Generation and Comprehension of Unambiguous Object Descriptions](https://mlsanity.com/paper/1511.02283/) (2015)
- [MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding](https://mlsanity.com/paper/2104.12763/) (2021)
- [DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting](https://mlsanity.com/paper/2112.01518/) (2021)
- [Image Segmentation Using Text and Image Prompts](https://mlsanity.com/paper/2112.10003/) (2021)
- [MAttNet: Modular Attention Network for Referring Expression Comprehension](https://mlsanity.com/paper/1801.08186/) (2018)
- [CRIS: CLIP-Driven Referring Image Segmentation](https://mlsanity.com/paper/2111.15174/) (2021)
- [ScribbleSup: Scribble-Supervised Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1604.05144/) (2016)
- [Cross-Modal Self-Attention Network for Referring Image Segmentation](https://mlsanity.com/paper/1904.04745/) (2019)
- [ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias](https://mlsanity.com/paper/2106.03348/) (2021)
- [Cut, Paste and Learn: Surprisingly Easy Synthesis for Instance Detection](https://mlsanity.com/paper/1708.01642/) (2017)
- [Multi-task Collaborative Network for Joint Referring Expression Comprehension and Segmentation](https://mlsanity.com/paper/2003.08813/) (2020)
- [MirrorGAN: Learning Text-to-image Generation by Redescription](https://mlsanity.com/paper/1903.05854/) (2019)
- [Real-Time High-Resolution Background Matting](https://mlsanity.com/paper/2012.07810/) (2020)
- [Deep Image Matting](https://mlsanity.com/paper/1703.03872/) (2017)
- [Referring Image Segmentation via Cross-Modal Progressive Comprehension](https://mlsanity.com/paper/2010.00514/) (2020)
- [MODNet: Real-Time Trimap-Free Portrait Matting via Objective Decomposition](https://mlsanity.com/paper/2011.11961/) (2020)
- [Recurrent Multimodal Interaction for Referring Image Segmentation](https://mlsanity.com/paper/1703.07939/) (2017)
- [PhraseCut: Language-based Image Segmentation in the Wild](https://mlsanity.com/paper/2008.01187/) (2020)
- [Empowering Things with Intelligence: A Survey of the Progress, Challenges, and Opportunities in Artificial Intelligence of Things](https://mlsanity.com/paper/2011.08612/) (2020)
- [Modeling Relationships in Referential Expressions with Compositional Modular Networks](https://mlsanity.com/paper/1611.09978/) (2016)

## Built on by (most cited first, 22 of 22)

- [Matting Anything](https://mlsanity.com/paper/2306.05399/) (2023)
- [CLAMP: Prompt-based Contrastive Learning for Connecting Language and Animal Pose](https://mlsanity.com/paper/2206.11752/) (2022)
- [RefSAM: Efficiently Adapting Segmenting Anything Model for Referring Video Object Segmentation](https://mlsanity.com/paper/2307.00997/) (2023)
- [Mobile Foundation Model as Firmware](https://mlsanity.com/paper/2308.14363/) (2023)
- [ZIM: Zero-Shot Image Matting for Anything](https://mlsanity.com/paper/2411.00626/) (2024)
- [Deep Image Matting: A Comprehensive Survey](https://mlsanity.com/paper/2304.04672/) (2023)
- [AlphaVAE: Unified End-to-End RGBA Image Reconstruction and Generation with Alpha-Aware Representation Learning](https://mlsanity.com/paper/2507.09308/) (2025)
- [SDMatte: Grafting Diffusion Models for Interactive Matting](https://mlsanity.com/paper/2508.00443/) (2025)
- [VideoMaMa: Mask-Guided Video Matting via Generative Prior](https://mlsanity.com/paper/2601.14255/) (2026)
- [Serverless Federated AUPRC Optimization for Multi-Party Collaborative Imbalanced Data Mining](https://mlsanity.com/paper/2308.03035/) (2023)
- [PartSeg: Few-shot Part Segmentation via Part-aware Prompt Learning](https://mlsanity.com/paper/2308.12757/) (2023)
- [In-Context Matting](https://mlsanity.com/paper/2403.15789/) (2024)
- [ParallelEdits: Efficient Multi-object Image Editing](https://mlsanity.com/paper/2406.00985/) (2024)
- [Towards Natural Image Matting in the Wild via Real-Scenario Prior](https://mlsanity.com/paper/2410.06593/) (2024)
- [Disentangled Pre-training for Image Matting](https://mlsanity.com/paper/2304.00784/) (2023)
- [DFIMat: Decoupled Flexible Interactive Matting in Multi-Person Scenarios](https://mlsanity.com/paper/2410.09788/) (2024)
- [VRMDiff: Text-Guided Video Referring Matting Generation of Diffusion](https://mlsanity.com/paper/2503.10678/) (2025)
- [Harnessing Vision-Language Pretrained Models with Temporal-Aware Adaptation for Referring Video Object Segmentation](https://mlsanity.com/paper/2405.10610/) (2024)
- [Synthetic Object Compositions for Scalable and Accurate Learning in Detection, Segmentation, and Grounding](https://mlsanity.com/paper/2510.09110/) (2025)
- [Parse Graph-Based Visual-Language Interaction for Human Pose Estimation](https://mlsanity.com/paper/2509.07385/) (2025)
- [SceneForge: Structured World Supervision from 3D Interventions](https://mlsanity.com/paper/2605.14399/) (2026)
- [SAM2Matting: Generalized Image and Video Matting](https://mlsanity.com/paper/2606.27339/) (2026)

## Abstract (arXiv)

Different from conventional image matting, which either requires user-defined scribbles/trimap to extract a specific foreground object or directly extracts all the foreground objects in the image indiscriminately, we introduce a new task named Referring Image Matting (RIM) in this paper, which aims to extract the meticulous alpha matte of the specific object that best matches the given natural language description, thus enabling a more natural and simpler instruction for image matting. First, we establish a large-scale challenging dataset RefMatte by designing a comprehensive image composition and expression generation engine to automatically produce high-quality images along with diverse text attributes based on public datasets. RefMatte consists of 230 object categories, 47,500 images, 118,749 expression-region entities, and 474,996 expressions. Additionally, we construct a real-world test set with 100 high-resolution natural images and manually annotate complex phrases to evaluate the out-of-domain generalization abilities of RIM methods. Furthermore, we present a novel baseline method CLIPMat for RIM, including a context-embedded prompt, a text-driven semantic pop-up, and a multi-level details extractor. Extensive experiments on RefMatte in both keyword and expression settings validate the superiority of CLIPMat over representative methods. We hope this work could provide novel insights into image matting and encourage more follow-up studies. The dataset, code and models are available at https://github.com/JizhiziLi/RIM.