# AnchorSeg: Language Grounded Query Banks for Reasoning Segmentation

arXiv: https://arxiv.org/abs/2604.18562 · HTML: https://mlsanity.com/paper/2604.18562/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 37 references that are in the map.
- First posted on arXiv 20 Apr 2026.

## Builds on (most cited first, 21 of 21)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Generation and Comprehension of Unambiguous Object Descriptions](https://mlsanity.com/paper/1511.02283/) (2015)
- [COCO-Stuff: Thing and Stuff Classes in Context](https://mlsanity.com/paper/1612.03716/) (2016)
- [NExT-GPT: Any-to-Any Multimodal LLM](https://mlsanity.com/paper/2309.05519/) (2023)
- [Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP](https://mlsanity.com/paper/2210.04150/) (2022)
- [Segment Everything Everywhere All at Once](https://mlsanity.com/paper/2304.06718/) (2023)
- [VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks](https://mlsanity.com/paper/2305.11175/) (2023)
- [MAttNet: Modular Attention Network for Referring Expression Comprehension](https://mlsanity.com/paper/1801.08186/) (2018)
- [GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest](https://mlsanity.com/paper/2307.03601/) (2023)
- [Generalized Decoding for Pixel, Image, and Language](https://mlsanity.com/paper/2212.11270/) (2022)
- [Vision-Language Transformer and Query Generation for Referring Segmentation](https://mlsanity.com/paper/2108.05565/) (2021)
- [Detect What You Can: Detecting and Representing Objects using Holistic Models and Body Parts](https://mlsanity.com/paper/1406.2031/) (2014)
- [PixelLM: Pixel Reasoning with Large Multimodal Model](https://mlsanity.com/paper/2312.02228/) (2023)
- [PACO: Parts and Attributes of Common Objects](https://mlsanity.com/paper/2301.01795/) (2023)
- [One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videos](https://mlsanity.com/paper/2409.19603/) (2024)
- [Locate then Segment: A Strong Pipeline for Referring Image Segmentation](https://mlsanity.com/paper/2103.16284/) (2021)
- [See, Say, and Segment: Teaching LMMs to Overcome False Premises](https://mlsanity.com/paper/2312.08366/) (2023)
- [Seg-ReSearch: Segmentation with Interleaved Reasoning and External Search](https://mlsanity.com/paper/2602.04454/) (2026)

## Built on by (most cited first, 1 of 1)

- [Skip the Talk, Re-Focus on Vision: Latent Reasoning for Reasoning Segmentation in Multimodal Large Language Models](https://mlsanity.com/paper/2609.30783/) (2026)

## Abstract (arXiv)

Reasoning segmentation requires models to ground complex, implicit textual queries into precise pixel-level masks. Existing approaches rely on a single segmentation token texttt<SEG>, whose hidden state implicitly encodes both semantic reasoning and spatial localization, limiting the model's ability to explicitly disentangle what to segment from where to segment. We introduce AnchorSeg, which reformulates reasoning segmentation as a structured conditional generation process over image tokens, conditioned on language grounded query banks. Instead of compressing all semantic reasoning and spatial localization into a single embedding, AnchorSeg constructs an ordered sequence of query banks: latent reasoning tokens that capture intermediate semantic states, and a segmentation anchor token that provides explicit spatial grounding. We model spatial conditioning as a factorized distribution over image tokens, where the anchor query determines localization signals while contextual queries provide semantic modulation. To bridge token-level predictions and pixel-level supervision, we propose Token--Mask Cycle Consistency (TMCC), a bidirectional training objective that enforces alignment across resolutions. By explicitly decoupling spatial grounding from semantic reasoning through structured language grounded query banks, AnchorSeg achieves state-of-the-art results on ReasonSeg test set (67.7% gIoU and 68.1% cIoU). All code and models are publicly available at https://github.com/rui-qian/AnchorSeg.