# MMR: A Large-scale Benchmark Dataset for Multi-target and Multi-granularity Reasoning Segmentation

arXiv: https://arxiv.org/abs/2503.13881 · HTML: https://mlsanity.com/paper/2503.13881/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 55 of its 62 references that are in the map.
- First posted on arXiv 18 Mar 2025.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)

## Builds on (most cited first, 30 of 55)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [Semantic Understanding of Scenes through the ADE20K Dataset](https://mlsanity.com/paper/1608.05442/) (2016)
- [Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic](https://mlsanity.com/paper/2306.15195/) (2023)
- [Generation and Comprehension of Unambiguous Object Descriptions](https://mlsanity.com/paper/1511.02283/) (2015)
- [COCO-Stuff: Thing and Stuff Classes in Context](https://mlsanity.com/paper/1612.03716/) (2016)
- [Playing for Data: Ground Truth from Computer Games](https://mlsanity.com/paper/1608.02192/) (2016)
- [Panoptic Segmentation](https://mlsanity.com/paper/1801.00868/) (2018)
- [LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention](https://mlsanity.com/paper/2303.16199/) (2023)
- [LISA: Reasoning Segmentation via Large Language Model](https://mlsanity.com/paper/2308.00692/) (2023)
- [Instruction Tuning with GPT-4](https://mlsanity.com/paper/2304.03277/) (2023)
- [LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model](https://mlsanity.com/paper/2304.15010/) (2023)
- [Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP](https://mlsanity.com/paper/2210.04150/) (2022)
- [Segment Everything Everywhere All at Once](https://mlsanity.com/paper/2304.06718/) (2023)
- [PartNet: A Large-scale Benchmark for Fine-grained and Hierarchical Part-level 3D Object Understanding](https://mlsanity.com/paper/1812.02713/) (2018)
- [Ferret: Refer and Ground Anything Anywhere at Any Granularity](https://mlsanity.com/paper/2310.07704/) (2023)
- [GLaMM: Pixel Grounding Large Multimodal Model](https://mlsanity.com/paper/2311.03356/) (2023)
- [CRIS: CLIP-Driven Referring Image Segmentation](https://mlsanity.com/paper/2111.15174/) (2021)
- [LAVT: Language-Aware Vision Transformer for Referring Image Segmentation](https://mlsanity.com/paper/2112.02244/) (2021)
- [GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest](https://mlsanity.com/paper/2307.03601/) (2023)

## Built on by (most cited first, 28 of 28)

- [UniPixel: Unified Object Referring and Segmentation for Pixel-Level Visual Reasoning](https://mlsanity.com/paper/2509.18094/) (2025)
- [From Perception to Cognition: A Survey of Vision-Language Interactive Reasoning in Multimodal Large Language Models](https://mlsanity.com/paper/2509.25373/) (2025)
- [Multimodal Referring Segmentation: A Survey](https://mlsanity.com/paper/2508.00265/) (2025)
- [Reasoning Segmentation for Images and Videos: A Survey](https://mlsanity.com/paper/2505.18816/) (2025)
- [SegEarth-R2: Towards Comprehensive Language-guided Segmentation for Remote Sensing Images](https://mlsanity.com/paper/2512.20013/) (2025)
- [Text4Seg++: Advancing Image Segmentation via Generative Language Modeling](https://mlsanity.com/paper/2509.06321/) (2025)
- [SAM3-I: Segment Anything with Instructions](https://mlsanity.com/paper/2512.04585/) (2025)
- [MediSee: Reasoning-based Pixel-level Perception in Medical Images](https://mlsanity.com/paper/2504.11008/) (2025)
- [Unlocking the Potential of MLLMs in Referring Expression Segmentation via a Light-weight Mask Decoder](https://mlsanity.com/paper/2508.04107/) (2025)
- [Better, Stronger, Faster: Tackling the Trilemma in MLLM-based Segmentation with Simultaneous Textual Mask Prediction](https://mlsanity.com/paper/2512.00395/) (2025)
- [ALTo: Adaptive-Length Tokenizer for Autoregressive Mask Generation](https://mlsanity.com/paper/2505.16495/) (2025)
- [Affogato: Open-Vocabulary Affordance Grounding with Automated Data Generation at Scale](https://mlsanity.com/paper/2506.12009/) (2025)
- [Qwen3-VL-Seg: Unlocking Open-World Referring Segmentation with Vision-Language Grounding](https://mlsanity.com/paper/2605.07141/) (2026)
- [AffordVLA: Injecting Affordance Representations into Vision-Language-Action Models via Implicit Feature Alignment](https://mlsanity.com/paper/2605.17517/) (2026)
- [HRSeg: High-Resolution Visual Perception and Enhancement for Reasoning Segmentation](https://mlsanity.com/paper/2507.12883/) (2025)
- [StAR: Segment Anything Reasoner](https://mlsanity.com/paper/2603.14382/) (2026)
- [SetCon: Towards Open-Ended Referring Segmentation via Set-Level Concept Prediction](https://mlsanity.com/paper/2605.20110/) (2026)
- [Re-purposing SAM into Efficient Visual Projectors for MLLM-Based Referring Image Segmentation](https://mlsanity.com/paper/2509.13676/) (2025)
- [MediRound: Multi-Round Entity-Level Reasoning Segmentation in Medical Images](https://mlsanity.com/paper/2511.12110/) (2025)
- [Vision Harnessing Agent for Open Ad-hoc Segmentation](https://mlsanity.com/paper/2605.19410/) (2026)
- [B-GRTO: Bootstrapped Group Relative Tool Optimization for Referring Segmentation](https://mlsanity.com/paper/2605.23500/) (2026)
- [Moment and Highlight Detection via MLLM Frame Segmentation](https://mlsanity.com/paper/2512.12246/) (2025)
- [Revisiting Salient Object Detection from an Observer-Centric Perspective](https://mlsanity.com/paper/2602.06369/) (2026)
- [From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Models](https://mlsanity.com/paper/2606.26196/) (2026)
- [GEAR-Seg: A Grounded Explainable Agent for Reasoning Segmentation and Data Engine](https://mlsanity.com/paper/2607.00544/) (2026)
- [Better, Stronger, Faster, and Broader: Structured All-Mask Prediction for MLLM-Based Segmentation](https://mlsanity.com/paper/2608.02791/) (2026)
- [0.5%>100%: Bidirectional Reciprocal Learning for Referring Image Segmentation](https://mlsanity.com/paper/2609.24510/) (2026)
- [Skip the Talk, Re-Focus on Vision: Latent Reasoning for Reasoning Segmentation in Multimodal Large Language Models](https://mlsanity.com/paper/2609.30783/) (2026)

## Abstract (arXiv)

The fusion of Large Language Models with vision models is pioneering new possibilities in user-interactive vision-language tasks. A notable application is reasoning segmentation, where models generate pixel-level segmentation masks by comprehending implicit meanings in human instructions. However, seamless human-AI interaction demands more than just object-level recognition; it requires understanding both objects and the functions of their detailed parts, particularly in multi-target scenarios. For example, when instructing a robot to textitturn on the TV", there could be various ways to accomplish this command. Recognizing multiple objects capable of turning on the TV, such as the TV itself or a remote control (multi-target), provides more flexible options and aids in finding the optimized scenario. Furthermore, understanding specific parts of these objects, like the TV's button or the remote's button (part-level), is important for completing the action. Unfortunately, current reasoning segmentation datasets predominantly focus on a single target object-level reasoning, which limits the detailed recognition of an object's parts in multi-target contexts. To address this gap, we construct a large-scale dataset called Multi-target and Multi-granularity Reasoning (MMR). MMR comprises 194K complex and implicit instructions that consider multi-target, object-level, and part-level aspects, based on pre-existing image-mask sets. This dataset supports diverse and context-aware interactions by hierarchically providing object and part information. Moreover, we propose a straightforward yet effective framework for multi-target, object-level, and part-level reasoning segmentation. Experimental results on MMR show that the proposed method can reason effectively in multi-target and multi-granularity scenarios, while the existing reasoning segmentation model still has room for improvement.