# Progressive Semantic-Guided Vision Transformer for Zero-Shot Learning

arXiv: https://arxiv.org/abs/2404.07713 · HTML: https://mlsanity.com/paper/2404.07713/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 58 references that are in the map.
- First posted on arXiv 11 Apr 2024.

## Builds on (most cited first, 30 of 44)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification](https://mlsanity.com/paper/2106.02034/) (2021)
- [Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations](https://mlsanity.com/paper/2202.07800/) (2022)
- [Feature Generating Networks for Zero-Shot Learning](https://mlsanity.com/paper/1712.00981/) (2017)
- [Zero-Shot Learning -- The Good, the Bad and the Ugly](https://mlsanity.com/paper/1703.04394/) (2017)
- [Local Relation Networks for Image Recognition](https://mlsanity.com/paper/1904.11491/) (2019)
- [Label-Embedding for Image Classification](https://mlsanity.com/paper/1503.08677/) (2015)
- [f-VAEGAN-D2: A Feature Generating Framework for Any-Shot Learning](https://mlsanity.com/paper/1903.10132/) (2019)
- [Attribute Prototype Network for Zero-Shot Learning](https://mlsanity.com/paper/2008.08290/) (2020)
- [Align Your Prompts: Test-Time Prompting with Distribution Alignment for Zero-Shot Generalization](https://mlsanity.com/paper/2311.01459/) (2023)
- [Counterfactual Zero-Shot and Open-Set Visual Recognition](https://mlsanity.com/paper/2103.00887/) (2021)
- [Contrastive Embedding for Generalized Zero-Shot Learning](https://mlsanity.com/paper/2103.16173/) (2021)
- [Goal-Oriented Gaze Estimation for Zero-Shot Learning](https://mlsanity.com/paper/2103.03433/) (2021)
- [FREE: Feature Refinement for Generalized Zero-Shot Learning](https://mlsanity.com/paper/2107.13807/) (2021)
- [Leveraging Seen and Unseen Semantic Relationships for Generative Zero-Shot Learning](https://mlsanity.com/paper/2007.09549/) (2020)
- [Semantic-Guided Multi-Attention Localization for Zero-Shot Learning](https://mlsanity.com/paper/1903.00502/) (2019)
- [HSVA: Hierarchical Semantic-Visual Adaptation for Zero-Shot Learning](https://mlsanity.com/paper/2109.15163/) (2021)
- [TransZero: Attribute-guided Transformer for Zero-Shot Learning](https://mlsanity.com/paper/2112.01683/) (2021)
- [I2MVFormer: Large Language Model Generated Multi-View Document Supervision for Zero-Shot Image Classification](https://mlsanity.com/paper/2212.02291/) (2022)
- [Improving Zero-Shot Generalization for CLIP with Synthesized Prompts](https://mlsanity.com/paper/2307.07397/) (2023)
- [MSDN: Mutually Semantic Distillation Network for Zero-Shot Learning](https://mlsanity.com/paper/2203.03137/) (2022)

## Built on by (most cited first, 17 of 17)

- [ZeroDiff: Solidified Visual-Semantic Correlation in Zero-Shot Learning](https://mlsanity.com/paper/2406.02929/) (2024)
- [SVIP: Semantically Contextualized Visual Patches for Zero-Shot Learning](https://mlsanity.com/paper/2503.10252/) (2025)
- [Interpretable Zero-Shot Learning with Locally-Aligned Vision-Language Model](https://mlsanity.com/paper/2506.23822/) (2025)
- [On the Element-Wise Representation and Reasoning in Zero-Shot Image Recognition: A Systematic Survey](https://mlsanity.com/paper/2408.04879/) (2024)
- [GenZSL: Generative Zero-Shot Learning Via Inductive Variational Autoencoder](https://mlsanity.com/paper/2505.11882/) (2025)
- [Visual and Semantic Prompt Collaboration for Generalized Zero-Shot Learning](https://mlsanity.com/paper/2503.23030/) (2025)
- [Visual-Semantic Graph Matching Net for Zero-Shot Learning](https://mlsanity.com/paper/2411.11351/) (2024)
- [MADS: Multi-Attribute Document Supervision for Zero-Shot Image Classification](https://mlsanity.com/paper/2503.06847/) (2025)
- [Enhancing Zero-Shot Image Recognition in Vision-Language Models through Human-like Concept Guidance](https://mlsanity.com/paper/2503.15886/) (2025)
- [Interpretable Zero-shot Learning with Infinite Class Concepts](https://mlsanity.com/paper/2505.03361/) (2025)
- [Semantic-Inductive Attribute Selection for Zero-Shot Learning](https://mlsanity.com/paper/2510.03260/) (2025)
- [Multi-Granularity Mutual Refinement Network for Zero-Shot Learning](https://mlsanity.com/paper/2511.08163/) (2025)
- [Fine-Grained Zero-Shot Learning with Attribute-Centric Representations](https://mlsanity.com/paper/2512.12219/) (2025)
- [Adversarial Robustness in Zero-Shot Learning:An Empirical Study on Class and Concept-Level Vulnerabilities](https://mlsanity.com/paper/2512.18651/) (2025)
- [ZeroDiff++: Substantial Unseen Visual-semantic Correlation in Zero-shot Learning](https://mlsanity.com/paper/2602.12401/) (2026)
- [Learning Adaptive and Visually Aligned Conditions for Generative Zero-Shot Learning](https://mlsanity.com/paper/2603.06281/) (2026)
- [Incentivizing Generative Zero-Shot Learning via Outcome-Reward Reinforcement Learning with Visual Cues](https://mlsanity.com/paper/2603.21138/) (2026)

## Abstract (arXiv)

Zero-shot learning (ZSL) recognizes the unseen classes by conducting visual-semantic interactions to transfer semantic knowledge from seen classes to unseen ones, supported by semantic information (e.g., attributes). However, existing ZSL methods simply extract visual features using a pre-trained network backbone (i.e., CNN or ViT), which fail to learn matched visual-semantic correspondences for representing semantic-related visual features as lacking of the guidance of semantic information, resulting in undesirable visual-semantic interactions. To tackle this issue, we propose a progressive semantic-guided vision transformer for zero-shot learning (dubbed ZSLViT). ZSLViT mainly considers two properties in the whole network: i) discover the semantic-related visual representations explicitly, and ii) discard the semantic-unrelated visual information. Specifically, we first introduce semantic-embedded token learning to improve the visual-semantic correspondences via semantic enhancement and discover the semantic-related visual tokens explicitly with semantic-guided token attention. Then, we fuse low semantic-visual correspondence visual tokens to discard the semantic-unrelated visual information for visual enhancement. These two operations are integrated into various encoders to progressively learn semantic-related visual representations for accurate visual-semantic interactions in ZSL. The extensive experiments show that our ZSLViT achieves significant performance gains on three popular benchmark datasets, i.e., CUB, SUN, and AWA2. Codes are available at: https://github.com/shiming-chen/ZSLViT .