mlsanityData to 14 Oct 2026

R2SM: Referring and Reasoning for Selective Masks

arXiv 2506.01795 ↗First posted 2 Jun 2025cs.CVcited by 0

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

We introduce a new task, Referring and Reasoning for Selective Masks (R2SM), which extends text-guided segmentation by incorporating mask-type selection driven by user intent. This task challenges vision-language models to determine whether to generate a modal (visible) or amodal (complete) segmentation mask based solely on natural language prompts. To support the R2SM task, we present the R2SM dataset, constructed by augmenting annotations of COCOA-cls, D2SA, and MUVA. The R2SM dataset consists of both modal and amodal text queries, each paired with the corresponding ground-truth mask, enabling model finetuning and evaluation for the ability to segment images as per user intent. Specifically, the task requires the model to interpret whether a given prompt refers to only the visible part of an object or to its complete shape, including occluded regions, and then produce the appropriate segmentation. For example, if a prompt explicitly requests the whole shape of a partially hidden object, the model is expected to output an amodal mask that completes the occluded parts. In contrast, prompts without explicit mention of hidden regions should generate standard modal masks. The R2SM benchmark provides a challenging and insightful testbed for advancing research in multimodal reasoning and intent-aware segmentation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 26 of its 29 references that are in the map. First posted on arXiv 2 Jun 2025.

Builds on

26 of 29 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understanding27 Jun
  2. 2023LISA++: An Improved Baseline for Reasoning Segmentation with Large Language Model28 Dec
  3. 2023PixelLM: Pixel Reasoning with Large Multimodal Model4 Dec
  4. 2023LISA: Reasoning Segmentation via Large Language Model1 Aug

Other references · 22

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,376
  3. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  4. 2016Modeling Context in Referring Expressionscited by 1,063
  5. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  6. 2023GLaMM: Pixel Grounding Large Multimodal Modelcited by 455
  7. 2021LAVT: Language-Aware Vision Transformer for Referring Image Segmentationcited by 348
  8. 2023GRES: Generalized Referring Expression Segmentationcited by 254
  9. 2023PACO: Parts and Attributes of Common Objectscited by 146
  10. 2021Referring Transformer: A One-step Approach to Multi-task Visual Groundingcited by 143
  11. 2022SeqTR: A Simple yet Universal Network for Visual Groundingcited by 139
  12. 2020PhraseCut: Language-based Image Segmentation in the Wildcited by 116
10 more
  1. 2015Semantic Amodal Segmentationcited by 103
  2. 2024PSALM: Pixelwise SegmentAtion with Large Multi-Modal Modelcited by 101
  3. 2024pix2gestalt: Amodal Segmentation by Synthesizing Wholescited by 70
  4. 2024SAM4MLLM: Enhance Multi-Modal Large Language Model for Referring Expression Segmentationcited by 69
  5. 2018Learning to See the Invisible: End-to-End Trainable Amodal Instance Segmentationcited by 62
  6. 2017SeGAN: Segmenting and Generating the Invisiblecited by 60
  7. 2023Amodal Ground Truth and Completion in the Wildcited by 45
  8. 2018MVTec D2S: Densely Segmented Supermarket Datasetcited by 20
  9. 2025Segment Anything, Even Occludedcited by 7
  10. 2025Unveiling the Invisible: Reasoning Complex Occlusions Amodally with AURAcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Refer to Any Segmentation Mask Group With Vision-Language Prompts
  2. 2026X2SAM: Any Segmentation in Images and Videos
  3. 2026Reason Twice: Segmentation via Candidate Discovery and Comparative Reasoning
  4. 2026GenSeg-R1: RL-Driven Vision-Language Grounding for Fine-Grained Referring Segmentation
  5. 2026CLIP-Guided SAM: Parameter-Efficient Semantic Conditioning for Promptable Segmentation
  6. 2026Toward Semantic-Agnostic and Shape-Aware Vision-Language Segmentation Models
  7. 2025Show or Tell? Effectively prompting Vision-Language Models for semantic segmentation
  8. 2025NOVO: Bridging LLaVA and SAM with Visual-only Prompts for Reasoning Segmentation