# HSVA: Hierarchical Semantic-Visual Adaptation for Zero-Shot Learning

arXiv: https://arxiv.org/abs/2109.15163 · HTML: https://mlsanity.com/paper/2109.15163/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 72 references that are in the map.
- First posted on arXiv 30 Sep 2021.

## Builds on (most cited first, 30 of 34)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks](https://mlsanity.com/paper/2004.06165/) (2020)
- [Maximum Classifier Discrepancy for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1712.02560/) (2017)
- [Generating Sentences from a Continuous Space](https://mlsanity.com/paper/1511.06349/) (2015)
- [Return of Frustratingly Easy Domain Adaptation](https://mlsanity.com/paper/1511.05547/) (2015)
- [Feature Generating Networks for Zero-Shot Learning](https://mlsanity.com/paper/1712.00981/) (2017)
- [Zero-Shot Learning -- The Good, the Bad and the Ugly](https://mlsanity.com/paper/1703.04394/) (2017)
- [Sliced Wasserstein Discrepancy for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1903.04064/) (2019)
- [Generalized Zero- and Few-Shot Learning via Aligned Variational Autoencoders](https://mlsanity.com/paper/1812.01784/) (2018)
- [Learning Deep Representations of Fine-grained Visual Descriptions](https://mlsanity.com/paper/1605.05395/) (2016)
- [Image Captioning: Transforming Objects into Words](https://mlsanity.com/paper/1906.05963/) (2019)
- [Evaluation of Output Embeddings for Fine-Grained Image Classification](https://mlsanity.com/paper/1409.8403/) (2014)
- [Label-Embedding for Image Classification](https://mlsanity.com/paper/1503.08677/) (2015)
- [f-VAEGAN-D2: A Feature Generating Framework for Any-Shot Learning](https://mlsanity.com/paper/1903.10132/) (2019)
- [Attribute Prototype Network for Zero-Shot Learning](https://mlsanity.com/paper/2008.08290/) (2020)
- [Synthesized Classifiers for Zero-Shot Learning](https://mlsanity.com/paper/1603.00550/) (2016)
- [Multi-modal Cycle-consistent Generalized Zero-Shot Learning](https://mlsanity.com/paper/1808.00136/) (2018)
- [Leveraging the Invariant Side of Generative Zero-Shot Learning](https://mlsanity.com/paper/1904.04092/) (2019)
- [Domain-aware Visual Bias Eliminating for Generalized Zero-Shot Learning](https://mlsanity.com/paper/2003.13261/) (2020)
- [Contextual-Relation Consistent Domain Adaptation for Semantic Segmentation](https://mlsanity.com/paper/2007.02424/) (2020)
- [FREE: Feature Refinement for Generalized Zero-Shot Learning](https://mlsanity.com/paper/2107.13807/) (2021)
- [Episode-based Prototype Generating Network for Zero-Shot Learning](https://mlsanity.com/paper/1909.03360/) (2019)
- [Transferable Contrastive Network for Generalized Zero-Shot Learning](https://mlsanity.com/paper/1908.05832/) (2019)
- [Leveraging Seen and Unseen Semantic Relationships for Generative Zero-Shot Learning](https://mlsanity.com/paper/2007.09549/) (2020)
- [Semantic-Guided Multi-Attention Localization for Zero-Shot Learning](https://mlsanity.com/paper/1903.00502/) (2019)
- [Learning Robust Visual-Semantic Embeddings](https://mlsanity.com/paper/1703.05908/) (2017)
- [Rethinking Zero-Shot Learning: A Conditional Visual Classification Perspective](https://mlsanity.com/paper/1909.05995/) (2019)
- [Zero-Shot Learning via Joint Latent Similarity Embedding](https://mlsanity.com/paper/1511.04512/) (2015)

## Built on by (most cited first, 30 of 57)

- [Model Adaptation: Historical Contrastive Learning for Unsupervised Domain Adaptation without Source Data](https://mlsanity.com/paper/2110.03374/) (2021)
- [CHiLS: Zero-Shot Image Classification with Hierarchical Label Sets](https://mlsanity.com/paper/2302.02551/) (2023)
- [Improving Zero-Shot Generalization for CLIP with Synthesized Prompts](https://mlsanity.com/paper/2307.07397/) (2023)
- [MSDN: Mutually Semantic Distillation Network for Zero-Shot Learning](https://mlsanity.com/paper/2203.03137/) (2022)
- [DUET: Cross-modal Semantic Grounding for Contrastive Zero-shot Learning](https://mlsanity.com/paper/2207.01328/) (2022)
- [Semantic-Aware Representation Blending for Multi-Label Image Recognition with Partial Labels](https://mlsanity.com/paper/2203.02172/) (2022)
- [TransZero++: Cross Attribute-Guided Transformer for Zero-Shot Learning](https://mlsanity.com/paper/2112.08643/) (2021)
- [Intra-Modal Proxy Learning for Zero-Shot Visual Categorization with CLIP](https://mlsanity.com/paper/2310.19752/) (2023)
- [Progressive Semantic-Visual Mutual Adaption for Generalized Zero-Shot Learning](https://mlsanity.com/paper/2303.15322/) (2023)
- [Zero-Shot Learning by Harnessing Adversarial Samples](https://mlsanity.com/paper/2308.00313/) (2023)
- [Evolving Semantic Prototype Improves Generative Zero-Shot Learning](https://mlsanity.com/paper/2306.06931/) (2023)
- [Progressive Semantic-Guided Vision Transformer for Zero-Shot Learning](https://mlsanity.com/paper/2404.07713/) (2024)
- [Visual-Augmented Dynamic Semantic Prototype for Generative Zero-Shot Learning](https://mlsanity.com/paper/2404.14808/) (2024)
- [CREST: Cross-modal Resonance through Evidential Deep Learning for Enhanced Zero-Shot Learning](https://mlsanity.com/paper/2404.09640/) (2024)
- [Open-Vocabulary Object Detection via Language Hierarchy](https://mlsanity.com/paper/2410.20371/) (2024)
- [Boosting Zero-shot Learning via Contrastive Optimization of Attribute Representations](https://mlsanity.com/paper/2207.03824/) (2022)
- [Rebalanced Zero-shot Learning](https://mlsanity.com/paper/2210.07031/) (2022)
- [Fine-Grained Zero-Shot Object Detection](https://mlsanity.com/paper/2507.10358/) (2025)
- [Deconstructed Generation-Based Zero-Shot Model](https://mlsanity.com/paper/2204.11280/) (2022)
- [Dual-Perspective Semantic-Aware Representation Blending for Multi-Label Image Recognition with Partial Labels](https://mlsanity.com/paper/2205.13092/) (2022)
- [EGANS: Evolutionary Generative Adversarial Network Search for Zero-Shot Learning](https://mlsanity.com/paper/2308.09915/) (2023)
- [Attribute-Aware Representation Rectification for Generalized Zero-Shot Learning](https://mlsanity.com/paper/2311.14750/) (2023)
- [SVIP: Semantically Contextualized Visual Patches for Zero-Shot Learning](https://mlsanity.com/paper/2503.10252/) (2025)
- [Interpretable Zero-Shot Learning with Locally-Aligned Vision-Language Model](https://mlsanity.com/paper/2506.23822/) (2025)
- [Exploring the Common Appearance-Boundary Adaptation for Nighttime Optical Flow](https://mlsanity.com/paper/2401.17642/) (2024)
- [Dual Expert Distillation Network for Generalized Zero-Shot Learning](https://mlsanity.com/paper/2404.16348/) (2024)
- [Adverse Weather Optical Flow: Cumulative Homogeneous-Heterogeneous Adaptation](https://mlsanity.com/paper/2409.17001/) (2024)
- [MaxSup: Overcoming Representation Collapse in Label Smoothing](https://mlsanity.com/paper/2502.15798/) (2025)
- [Visual and Semantic Prompt Collaboration for Generalized Zero-Shot Learning](https://mlsanity.com/paper/2503.23030/) (2025)
- [Unsupervised Hierarchical Domain Adaptation for Adverse Weather Optical Flow](https://mlsanity.com/paper/2303.13761/) (2023)

## Abstract (arXiv)

Zero-shot learning (ZSL) tackles the unseen class recognition problem, transferring semantic knowledge from seen classes to unseen ones. Typically, to guarantee desirable knowledge transfer, a common (latent) space is adopted for associating the visual and semantic domains in ZSL. However, existing common space learning methods align the semantic and visual domains by merely mitigating distribution disagreement through one-step adaptation. This strategy is usually ineffective due to the heterogeneous nature of the feature representations in the two domains, which intrinsically contain both distribution and structure variations. To address this and advance ZSL, we propose a novel hierarchical semantic-visual adaptation (HSVA) framework. Specifically, HSVA aligns the semantic and visual domains by adopting a hierarchical two-step adaptation, i.e., structure adaptation and distribution adaptation. In the structure adaptation step, we take two task-specific encoders to encode the source data (visual domain) and the target data (semantic domain) into a structure-aligned common space. To this end, a supervised adversarial discrepancy (SAD) module is proposed to adversarially minimize the discrepancy between the predictions of two task-specific classifiers, thus making the visual and semantic feature manifolds more closely aligned. In the distribution adaptation step, we directly minimize the Wasserstein distance between the latent multivariate Gaussian distributions to align the visual and semantic distributions using a common encoder. Finally, the structure and distribution adaptation are derived in a unified framework under two partially-aligned variational autoencoders. Extensive experiments on four benchmark datasets demonstrate that HSVA achieves superior performance on both conventional and generalized ZSL. The code is available at urlhttps://github.com/shiming-chen/HSVA .