# SIGN: Spatial-information Incorporated Generative Network for Generalized Zero-shot Semantic Segmentation

arXiv: https://arxiv.org/abs/2108.12517 · HTML: https://mlsanity.com/paper/2108.12517/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 64 references that are in the map.
- First posted on arXiv 27 Aug 2021.

## Builds on (most cited first, 30 of 40)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [COCO-Stuff: Thing and Stuff Classes in Context](https://mlsanity.com/paper/1612.03716/) (2016)
- [Improved Knowledge Distillation via Teacher Assistant](https://mlsanity.com/paper/1902.03393/) (2019)
- [Generative Moment Matching Networks](https://mlsanity.com/paper/1502.02761/) (2015)
- [Feature Generating Networks for Zero-Shot Learning](https://mlsanity.com/paper/1712.00981/) (2017)
- [Zero-Shot Learning -- The Good, the Bad and the Ugly](https://mlsanity.com/paper/1703.04394/) (2017)
- [Zero-Shot Semantic Segmentation](https://mlsanity.com/paper/1906.00817/) (2019)
- [Zero-Shot Learning Through Cross-Modal Transfer](https://mlsanity.com/paper/1301.3666/) (2013)
- [Zero-Shot Object Detection](https://mlsanity.com/paper/1804.04340/) (2018)
- [Evaluation of Output Embeddings for Fine-Grained Image Classification](https://mlsanity.com/paper/1409.8403/) (2014)
- [Label-Embedding for Image Classification](https://mlsanity.com/paper/1503.08677/) (2015)
- [Zero-shot Recognition via Semantic Embeddings and Knowledge Graphs](https://mlsanity.com/paper/1803.08035/) (2018)
- [Zero-Shot Learning by Convex Combination of Semantic Embeddings](https://mlsanity.com/paper/1312.5650/) (2013)
- [Latent Embeddings for Zero-shot Classification](https://mlsanity.com/paper/1603.08895/) (2016)
- [Context-aware Feature Generation for Zero-shot Semantic Segmentation](https://mlsanity.com/paper/2008.06893/) (2020)
- [Rethinking Knowledge Graph Propagation for Zero-Shot Learning](https://mlsanity.com/paper/1805.11724/) (2018)

## Built on by (most cited first, 30 of 41)

- [Decoupling Zero-Shot Semantic Segmentation](https://mlsanity.com/paper/2112.07910/) (2021)
- [CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigation](https://mlsanity.com/paper/2203.10421/) (2022)
- [ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentation](https://mlsanity.com/paper/2212.03588/) (2022)
- [DiffuMask: Synthesizing Images with Pixel-level Annotations for Semantic Segmentation Using Diffusion Models](https://mlsanity.com/paper/2303.11681/) (2023)
- [DatasetDM: Synthesizing Data with Perception Annotations Using Diffusion Models](https://mlsanity.com/paper/2308.06160/) (2023)
- [Learning Open-vocabulary Semantic Segmentation Models From Natural Language Supervision](https://mlsanity.com/paper/2301.09121/) (2023)
- [Open-Vocabulary Instance Segmentation via Robust Cross-Modal Pseudo-Labeling](https://mlsanity.com/paper/2111.12698/) (2021)
- [Open-vocabulary Object Segmentation with Diffusion Models](https://mlsanity.com/paper/2301.05221/) (2023)
- [Open-world Semantic Segmentation via Contrasting and Clustering Vision-Language Embedding](https://mlsanity.com/paper/2207.08455/) (2022)
- [Diffusion Models for Open-Vocabulary Segmentation](https://mlsanity.com/paper/2306.09316/) (2023)
- [SATR: Zero-Shot Semantic Segmentation of 3D Shapes](https://mlsanity.com/paper/2304.04909/) (2023)
- [A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Future](https://mlsanity.com/paper/2307.09220/) (2023)
- [Primitive Generation and Semantic-related Alignment for Universal Zero-Shot Segmentation](https://mlsanity.com/paper/2306.11087/) (2023)
- [See More and Know More: Zero-shot Point Cloud Segmentation via Multi-modal Visual Data](https://mlsanity.com/paper/2307.10782/) (2023)
- [Cascade-CLIP: Cascaded Vision-Language Embeddings Alignment for Zero-Shot Semantic Segmentation](https://mlsanity.com/paper/2406.00670/) (2024)
- [Semantic-Promoted Debiasing and Background Disambiguation for Zero-Shot Instance Segmentation](https://mlsanity.com/paper/2305.13173/) (2023)
- [IFSeg: Image-free Semantic Segmentation via Vision-Language Model](https://mlsanity.com/paper/2303.14396/) (2023)
- [Towards Training-free Open-world Segmentation via Image Prompt Foundation Models](https://mlsanity.com/paper/2310.10912/) (2023)
- [TagCLIP: Improving Discrimination Ability of Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2304.07547/) (2023)
- [IPO: Interpretable Prompt Optimization for Vision-Language Models](https://mlsanity.com/paper/2410.15397/) (2024)
- [Spectral Prompt Tuning:Unveiling Unseen Classes for Zero-Shot Semantic Segmentation](https://mlsanity.com/paper/2312.12754/) (2023)
- [MVP-SEG: Multi-View Prompt Learning for Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2304.06957/) (2023)
- [Vision-Language Model for Object Detection and Segmentation: A Review and Evaluation](https://mlsanity.com/paper/2504.09480/) (2025)
- [CLIP Is Also a Good Teacher: A New Learning Framework for Inductive Zero-shot Semantic Segmentation](https://mlsanity.com/paper/2310.02296/) (2023)
- [Language-Driven Visual Consensus for Zero-Shot Semantic Segmentation](https://mlsanity.com/paper/2403.08426/) (2024)
- [Instance-level Few-shot Learning with Class Hierarchy Mining](https://mlsanity.com/paper/2304.07459/) (2023)
- [AlignZeg: Mitigating Objective Misalignment for Zero-shot Semantic Segmentation](https://mlsanity.com/paper/2404.05667/) (2024)
- [SeeDiff: Off-the-Shelf Seeded Mask Generation from Diffusion Models](https://mlsanity.com/paper/2507.19808/) (2025)
- [Target Refocusing via Attention Redistribution for Open-Vocabulary Semantic Segmentation: An Explainability Perspective](https://mlsanity.com/paper/2511.16170/) (2025)
- [A Language-Guided Benchmark for Weakly Supervised Open Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2302.14163/) (2023)

## Abstract (arXiv)

Unlike conventional zero-shot classification, zero-shot semantic segmentation predicts a class label at the pixel level instead of the image level. When solving zero-shot semantic segmentation problems, the need for pixel-level prediction with surrounding context motivates us to incorporate spatial information using positional encoding. We improve standard positional encoding by introducing the concept of Relative Positional Encoding, which integrates spatial information at the feature level and can handle arbitrary image sizes. Furthermore, while self-training is widely used in zero-shot semantic segmentation to generate pseudo-labels, we propose a new knowledge-distillation-inspired self-training strategy, namely Annealed Self-Training, which can automatically assign different importance to pseudo-labels to improve performance. We systematically study the proposed Relative Positional Encoding and Annealed Self-Training in a comprehensive experimental evaluation, and our empirical results confirm the effectiveness of our method on three benchmark datasets.