# Unlocking the Potential of MLLMs in Referring Expression Segmentation via a Light-weight Mask Decoder

arXiv: https://arxiv.org/abs/2508.04107 · HTML: https://mlsanity.com/paper/2508.04107/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 23 references that are in the map.
- First posted on arXiv 6 Aug 2025.

## Builds on (most cited first, 19 of 19)

- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://mlsanity.com/paper/2412.05271/) (2024)
- [Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network](https://mlsanity.com/paper/1609.05158/) (2016)
- [Modeling Context in Referring Expressions](https://mlsanity.com/paper/1608.00272/) (2016)
- [How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites](https://mlsanity.com/paper/2404.16821/) (2024)
- [Generation and Comprehension of Unambiguous Object Descriptions](https://mlsanity.com/paper/1511.02283/) (2015)
- [VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks](https://mlsanity.com/paper/2406.08394/) (2024)
- [Encoder Fusion Network with Co-Attention Embedding for Referring Image Segmentation](https://mlsanity.com/paper/2105.01839/) (2021)
- [PSALM: Pixelwise SegmentAtion with Large Multi-Modal Model](https://mlsanity.com/paper/2403.14598/) (2024)
- [Locate then Segment: A Strong Pipeline for Referring Image Segmentation](https://mlsanity.com/paper/2103.16284/) (2021)
- [Text4Seg: Reimagining Image Segmentation as Text Generation](https://mlsanity.com/paper/2410.09855/) (2024)
- [Learning to Upsample by Learning to Sample](https://mlsanity.com/paper/2308.15085/) (2023)
- [LaSagnA: Language-based Segmentation Assistant for Complex Queries](https://mlsanity.com/paper/2404.08506/) (2024)
- [Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perception](https://mlsanity.com/paper/2403.02969/) (2024)
- [MMR: A Large-scale Benchmark Dataset for Multi-target and Multi-granularity Reasoning Segmentation](https://mlsanity.com/paper/2503.13881/) (2025)
- [UFO: A Unified Approach to Fine-grained Visual Perception via Open-ended Language Interface](https://mlsanity.com/paper/2503.01342/) (2025)
- [The Devil is in Temporal Token: High Quality Video Reasoning Segmentation](https://mlsanity.com/paper/2501.08549/) (2025)
- [HiMTok: Learning Hierarchical Mask Tokens for Image Segmentation with Large Multimodal Model](https://mlsanity.com/paper/2503.13026/) (2025)
- [Progressive Language-guided Visual Learning for Multi-Task Visual Grounding](https://mlsanity.com/paper/2504.16145/) (2025)

## Built on by (most cited first, 8 of 8)

- [SAMTok: Representing Any Mask with Two Words](https://mlsanity.com/paper/2601.16093/) (2026)
- [Qwen3-VL-Seg: Unlocking Open-World Referring Segmentation with Vision-Language Grounding](https://mlsanity.com/paper/2605.07141/) (2026)
- [VPTracker: Global Vision-Language Tracking via Visual Prompt](https://mlsanity.com/paper/2512.22799/) (2025)
- [SetCon: Towards Open-Ended Referring Segmentation via Set-Level Concept Prediction](https://mlsanity.com/paper/2605.20110/) (2026)
- [Actor as Its Own Critic: Unifying Region Understanding and Localization via CycleGRPO](https://mlsanity.com/paper/2607.11581/) (2026)
- [MIRAGE: Benchmarking and Aligning Multi-Instance Image Editing](https://mlsanity.com/paper/2604.05180/) (2026)
- [PANORAMA: Panoptic Grounded Captioning via Mask Proposal Selection](https://mlsanity.com/paper/2609.19143/) (2026)
- [Seg2Click: Learning Pixel-Level Action Masks for GUI Visual Grounding](https://mlsanity.com/paper/2610.16234/) (2026)

## Abstract (arXiv)

Reference Expression Segmentation (RES) aims to segment image regions specified by referring expressions and has become popular with the rise of multimodal large models (MLLMs). While MLLMs excel in semantic understanding, their token-generation paradigm struggles with pixel-level dense prediction. Existing RES methods either couple MLLMs with the parameter-heavy Segment Anything Model (SAM) with 632M network parameters or adopt SAM-free lightweight pipelines that sacrifice accuracy. To address the trade-off between performance and cost, we specifically propose MLLMSeg, a novel framework that fully exploits the inherent visual detail features encoded in the MLLM vision encoder without introducing an extra visual encoder. Besides, we propose a detail-enhanced and semantic-consistent feature fusion module (DSFF) that fully integrates the detail-related visual feature with the semantic-related feature output by the large language model (LLM) of MLLM. Finally, we establish a light-weight mask decoder with only 34M network parameters that optimally leverages detailed spatial features from the visual encoder and semantic features from the LLM to achieve precise mask prediction. Extensive experiments demonstrate that our method generally surpasses both SAM-based and SAM-free competitors, striking a better balance between performance and cost. Code is available at https://github.com/jcwang0602/MLLMSeg.