# AttrSeg: Open-Vocabulary Semantic Segmentation via Attribute Decomposition-Aggregation

arXiv: https://arxiv.org/abs/2309.00096 · HTML: https://mlsanity.com/paper/2309.00096/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 61 references that are in the map.
- First posted on arXiv 31 Aug 2023.

## Builds on (most cited first, 30 of 43)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [Object-Centric Learning with Slot Attention](https://mlsanity.com/paper/2006.15055/) (2020)
- [COCO-Stuff: Thing and Stuff Classes in Context](https://mlsanity.com/paper/1612.03716/) (2016)
- [Florence: A New Foundation Model for Computer Vision](https://mlsanity.com/paper/2111.11432/) (2021)
- [Language-driven Semantic Segmentation](https://mlsanity.com/paper/2201.03546/) (2022)
- [FILIP: Fine-grained Interactive Language-Image Pre-Training](https://mlsanity.com/paper/2111.07783/) (2021)
- [VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding](https://mlsanity.com/paper/2109.14084/) (2021)
- [Extract Free Dense Labels from CLIP](https://mlsanity.com/paper/2112.01071/) (2021)
- [LiT: Zero-Shot Transfer with Locked-image text Tuning](https://mlsanity.com/paper/2111.07991/) (2021)
- [Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP](https://mlsanity.com/paper/2210.04150/) (2022)
- [Scaling Open-Vocabulary Image Segmentation with Image-Level Labels](https://mlsanity.com/paper/2112.12143/) (2021)
- [Visual Classification via Description from Large Language Models](https://mlsanity.com/paper/2210.07183/) (2022)
- [Prompting Visual-Language Models for Efficient Video Understanding](https://mlsanity.com/paper/2112.04478/) (2021)
- [What does a platypus look like? Generating customized prompts for zero-shot image classification](https://mlsanity.com/paper/2209.03320/) (2022)
- [Zero-Shot Semantic Segmentation](https://mlsanity.com/paper/1906.00817/) (2019)
- [CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2303.11797/) (2023)
- [Decoupling Zero-Shot Semantic Segmentation](https://mlsanity.com/paper/2112.07910/) (2021)
- [Semantic Autoencoder for Zero-Shot Learning](https://mlsanity.com/paper/1704.08345/) (2017)
- [Learning to Predict Visual Attributes in the Wild](https://mlsanity.com/paper/2106.09707/) (2021)
- [Context-aware Feature Generation for Zero-shot Semantic Segmentation](https://mlsanity.com/paper/2008.06893/) (2020)
- [CHiLS: Zero-Shot Image Classification with Hierarchical Label Sets](https://mlsanity.com/paper/2302.02551/) (2023)
- [Complementary Patch for Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/2108.03852/) (2021)
- [DiffusionSeg: Adapting Diffusion Towards Unsupervised Object Discovery](https://mlsanity.com/paper/2303.09813/) (2023)
- [Exploiting a Joint Embedding Space for Generalized Zero-Shot Semantic Segmentation](https://mlsanity.com/paper/2108.06536/) (2021)
- [Annotation-free Audio-Visual Segmentation](https://mlsanity.com/paper/2305.11019/) (2023)

## Built on by (most cited first, 29 of 29)

- [LLaFS: When Large Language Models Meet Few-Shot Segmentation](https://mlsanity.com/paper/2311.16926/) (2023)
- [ReMamber: Referring Image Segmentation with Mamba Twister](https://mlsanity.com/paper/2403.17839/) (2024)
- [Open-YOLO 3D: Towards Fast and Accurate Open-Vocabulary 3D Instance Segmentation](https://mlsanity.com/paper/2406.02548/) (2024)
- [FOLDER: Accelerating Multi-modal Large Language Models with Enhanced Performance](https://mlsanity.com/paper/2501.02430/) (2025)
- [Uncovering Prototypical Knowledge for Weakly Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2310.19001/) (2023)
- [Open Panoramic Segmentation](https://mlsanity.com/paper/2407.02685/) (2024)
- [Turbo: Informativity-Driven Acceleration Plug-In for Vision-Language Large Models](https://mlsanity.com/paper/2407.11717/) (2024)
- [Advancing Myopia To Holism: Fully Contrastive Language-Image Pre-training](https://mlsanity.com/paper/2412.00440/) (2024)
- [Turbo: Informativity-Driven Acceleration Plug-In for Vision-Language Models](https://mlsanity.com/paper/2312.07408/) (2023)
- [DENOISER: Rethinking the Robustness for Open-Vocabulary Action Recognition](https://mlsanity.com/paper/2404.14890/) (2024)
- [Renovating Names in Open-Vocabulary Segmentation Benchmarks](https://mlsanity.com/paper/2403.09593/) (2024)
- [Black Sheep in the Herd: Playing with Spuriously Correlated Attributes for Vision-Language Recognition](https://mlsanity.com/paper/2502.15809/) (2025)
- [Decouple before Align: Visual Disentanglement Enhances Prompt Tuning](https://mlsanity.com/paper/2508.00395/) (2025)
- [DSV-LFS: Unifying LLM-Driven Semantic Cues with Visual Features for Robust Few-Shot Segmentation](https://mlsanity.com/paper/2503.04006/) (2025)
- [G4Seg: Generation for Inexact Segmentation Refinement with Diffusion Models](https://mlsanity.com/paper/2506.01539/) (2025)
- [Storyboard guided Alignment for Fine-grained Video Action Recognition](https://mlsanity.com/paper/2410.14238/) (2024)
- [From Open-Vocabulary to Vocabulary-Free Semantic Segmentation](https://mlsanity.com/paper/2502.11891/) (2025)
- [MoMa: Modulating Mamba for Adapting Image Foundation Models to Video Recognition](https://mlsanity.com/paper/2506.23283/) (2025)
- [SaFiRe: Saccade-Fixation Reiteration with Mamba for Referring Image Segmentation](https://mlsanity.com/paper/2510.10160/) (2025)
- [Explore More, Learn Better: Parallel MLLM Embeddings under Mutual Information Minimization](https://mlsanity.com/paper/2511.01588/) (2025)
- [GenMask: Adapting DiT for Segmentation via Direct Mask Generation](https://mlsanity.com/paper/2603.23906/) (2026)
- [Squeeze Out Tokens from Sample for Finer-Grained Data Governance](https://mlsanity.com/paper/2503.14559/) (2025)
- [What You Perceive Is What You Conceive: A Cognition-Inspired Framework for Open Vocabulary Image Segmentation](https://mlsanity.com/paper/2505.19569/) (2025)
- [Wave-Particle (Continuous-Discrete) Dualistic Visual Tokenization for Unified Understanding and Generation](https://mlsanity.com/paper/2511.01593/) (2025)
- [TransText: Alpha-as-RGB Representation for Transparent Text Animation](https://mlsanity.com/paper/2603.17944/) (2026)
- [TFANet: Three-Stage Image-Text Feature Alignment Network for Robust Referring Image Segmentation](https://mlsanity.com/paper/2509.13070/) (2025)
- [Hierarchical Spatial and Channel Aggregation for Cross-domain Few-shot Segmentation](https://mlsanity.com/paper/2606.24296/) (2026)
- [TRACE-Bench: Decomposing and Diagnosing Multi-Reference Image Generation](https://mlsanity.com/paper/2608.16765/) (2026)
- [Preference-Guided Adaptation for Open-Vocabulary Semantic Segmentation via Prompt Disagreement](https://mlsanity.com/paper/2609.34528/) (2026)

## Abstract (arXiv)

Open-vocabulary semantic segmentation is a challenging task that requires segmenting novel object categories at inference time. Recent studies have explored vision-language pre-training to handle this task, but suffer from unrealistic assumptions in practical scenarios, i.e., low-quality textual category names. For example, this paradigm assumes that new textual categories will be accurately and completely provided, and exist in lexicons during pre-training. However, exceptions often happen when encountering ambiguity for brief or incomplete names, new words that are not present in the pre-trained lexicons, and difficult-to-describe categories for users. To address these issues, this work proposes a novel attribute decomposition-aggregation framework, AttrSeg, inspired by human cognition in understanding new concepts. Specifically, in the decomposition stage, we decouple class names into diverse attribute descriptions to complement semantic contexts from multiple perspectives. Two attribute construction strategies are designed: using large language models for common categories, and involving manually labeling for human-invented categories. In the aggregation stage, we group diverse attributes into an integrated global description, to form a discriminative classifier that distinguishes the target object from others. One hierarchical aggregation architecture is further proposed to achieve multi-level aggregations, leveraging the meticulously designed clustering module. The final results are obtained by computing the similarity between aggregated attributes and images embeddings. To evaluate the effectiveness, we annotate three types of datasets with attribute descriptions, and conduct extensive experiments and ablation studies. The results show the superior performance of attribute decomposition-aggregation.