# What You Perceive Is What You Conceive: A Cognition-Inspired Framework for Open Vocabulary Image Segmentation

arXiv: https://arxiv.org/abs/2505.19569 · HTML: https://mlsanity.com/paper/2505.19569/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 56 references that are in the map.
- First posted on arXiv 26 May 2025.

## Builds on (most cited first, 30 of 34)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts](https://mlsanity.com/paper/2102.08981/) (2021)
- [Panoptic Segmentation](https://mlsanity.com/paper/1801.00868/) (2018)
- [Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP](https://mlsanity.com/paper/2210.04150/) (2022)
- [GroupViT: Semantic Segmentation Emerges from Text Supervision](https://mlsanity.com/paper/2202.11094/) (2022)
- [MAttNet: Modular Attention Network for Referring Expression Comprehension](https://mlsanity.com/paper/1801.08186/) (2018)
- [Panoptic-DeepLab](https://mlsanity.com/paper/1910.04751/) (2019)
- [MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers](https://mlsanity.com/paper/2012.00759/) (2020)
- [Side Adapter Network for Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2302.12242/) (2023)
- [Generalized Decoding for Pixel, Image, and Language](https://mlsanity.com/paper/2212.11270/) (2022)
- [Segmentation from Natural Language Expressions](https://mlsanity.com/paper/1603.06180/) (2016)
- [Zero-Shot Semantic Segmentation](https://mlsanity.com/paper/1906.00817/) (2019)
- [YFCC100M: The New Data in Multimedia Research](https://mlsanity.com/paper/1503.01817/) (2015)
- [CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2303.11797/) (2023)
- [Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIP](https://mlsanity.com/paper/2308.02487/) (2023)
- [Open-Vocabulary Universal Image Segmentation with MaskCLIP](https://mlsanity.com/paper/2208.08984/) (2022)
- [ReSTR: Convolution-free Referring Image Segmentation Using Transformers](https://mlsanity.com/paper/2203.16768/) (2022)
- [Panoptic SegFormer: Delving Deeper into Panoptic Segmentation with Transformers](https://mlsanity.com/paper/2109.03814/) (2021)
- [FreeSeg: Unified, Universal and Open-Vocabulary Image Segmentation](https://mlsanity.com/paper/2303.17225/) (2023)
- [Collaborative Vision-Text Representation Optimizing for Open-Vocabulary Segmentation](https://mlsanity.com/paper/2408.00744/) (2024)
- [You Only Segment Once: Towards Real-Time Panoptic Segmentation](https://mlsanity.com/paper/2303.14651/) (2023)
- [Open-vocabulary Panoptic Segmentation with Embedding Modulation](https://mlsanity.com/paper/2303.11324/) (2023)
- [Open-Vocabulary Semantic Segmentation with Image Embedding Balancing](https://mlsanity.com/paper/2406.09829/) (2024)

## Built on by (most cited first, 2 of 2)

- [Active-SAOOD: Active Sparsely Annotated Oriented Object Detection in Remote Sensing Images](https://mlsanity.com/paper/2605.10162/) (2026)
- [Referring Industrial Anomaly Segmentation](https://mlsanity.com/paper/2602.03673/) (2026)

## Abstract (arXiv)

Open vocabulary image segmentation tackles the challenge of recognizing dynamically adjustable, predefined novel categories at inference time by leveraging vision-language alignment. However, existing paradigms typically perform class-agnostic region segmentation followed by category matching, which deviates from the human visual system's process of recognizing objects based on semantic concepts, leading to poor alignment between region segmentation and target concepts. To bridge this gap, we propose a novel Cognition-Inspired Framework for open vocabulary image segmentation that emulates the human visual recognition process: first forming a conceptual understanding of an object, then perceiving its spatial extent. The framework consists of three core components: (1) A Generative Vision-Language Model (G-VLM) that mimics human cognition by generating object concepts to provide semantic guidance for region segmentation. (2) A Concept-Aware Visual Enhancer Module that fuses textual concept features with global visual representations, enabling adaptive visual perception based on target concepts. (3) A Cognition-Inspired Decoder that integrates local instance features with G-VLM-provided semantic cues, allowing selective classification over a subset of relevant categories. Extensive experiments demonstrate that our framework achieves significant improvements, reaching 27.2 PQ, 17.0 mAP, and 35.3 mIoU on A-150. It further attains 56.2, 28.2, 15.4, 59.2, 18.7, and 95.8 mIoU on Cityscapes, Mapillary Vistas, A-847, PC-59, PC-459, and PAS-20, respectively. In addition, our framework supports vocabulary-free segmentation, offering enhanced flexibility in recognizing unseen categories. Code will be public.