# Hybrid Routing Transformer for Zero-Shot Learning

arXiv: https://arxiv.org/abs/2203.15310 · HTML: https://mlsanity.com/paper/2203.15310/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 70 references that are in the map.
- First posted on arXiv 29 Mar 2022.

## Builds on (most cited first, 30 of 39)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [NICE: Non-linear Independent Components Estimation](https://mlsanity.com/paper/1410.8516/) (2014)
- [Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Ugly](https://mlsanity.com/paper/1707.00600/) (2017)
- [Feature Generating Networks for Zero-Shot Learning](https://mlsanity.com/paper/1712.00981/) (2017)
- [Zero-Shot Learning -- The Good, the Bad and the Ugly](https://mlsanity.com/paper/1703.04394/) (2017)
- [Evaluation of Output Embeddings for Fine-Grained Image Classification](https://mlsanity.com/paper/1409.8403/) (2014)
- [Label-Embedding for Image Classification](https://mlsanity.com/paper/1503.08677/) (2015)
- [f-VAEGAN-D2: A Feature Generating Framework for Any-Shot Learning](https://mlsanity.com/paper/1903.10132/) (2019)
- [Attribute Prototype Network for Zero-Shot Learning](https://mlsanity.com/paper/2008.08290/) (2020)
- [Zero-Shot Learning by Convex Combination of Semantic Embeddings](https://mlsanity.com/paper/1312.5650/) (2013)
- [Semantic Autoencoder for Zero-Shot Learning](https://mlsanity.com/paper/1704.08345/) (2017)
- [Latent Embeddings for Zero-shot Classification](https://mlsanity.com/paper/1603.08895/) (2016)
- [Generalized Zero-Shot Learning via Synthesized Examples](https://mlsanity.com/paper/1712.03878/) (2017)
- [Synthesized Classifiers for Zero-Shot Learning](https://mlsanity.com/paper/1603.00550/) (2016)
- [Counterfactual Zero-Shot and Open-Set Visual Recognition](https://mlsanity.com/paper/2103.00887/) (2021)
- [Multi-modal Cycle-consistent Generalized Zero-Shot Learning](https://mlsanity.com/paper/1808.00136/) (2018)
- [Weakly Supervised Object Localization and Detection: A Survey](https://mlsanity.com/paper/2104.07918/) (2021)
- [Zero-Shot Learning via Semantic Similarity Embedding](https://mlsanity.com/paper/1509.04767/) (2015)
- [Zero-Shot Visual Recognition using Semantics-Preserving Adversarial Embedding Networks](https://mlsanity.com/paper/1712.01928/) (2017)
- [Episode-based Prototype Generating Network for Zero-Shot Learning](https://mlsanity.com/paper/1909.03360/) (2019)
- [Semantic-Guided Multi-Attention Localization for Zero-Shot Learning](https://mlsanity.com/paper/1903.00502/) (2019)
- [Preserving Semantic Relations for Zero-Shot Learning](https://mlsanity.com/paper/1803.03049/) (2018)

## Built on by (most cited first, 20 of 20)

- [A Closer Look at the Explainability of Contrastive Language-Image Pre-training](https://mlsanity.com/paper/2304.05653/) (2023)
- [Knowledge-Enhanced Dual-stream Zero-shot Composed Image Retrieval](https://mlsanity.com/paper/2403.16005/) (2024)
- [Transformer for Object Re-Identification: A Survey](https://mlsanity.com/paper/2401.06960/) (2024)
- [GP-NeRF: Generalized Perception NeRF for Context-Aware 3D Scene Understanding](https://mlsanity.com/paper/2311.11863/) (2023)
- [Fine-Grained Zero-Shot Learning: Advances, Challenges, and Prospects](https://mlsanity.com/paper/2401.17766/) (2024)
- [MV-CLIP: Multi-View CLIP for Zero-shot 3D Shape Recognition](https://mlsanity.com/paper/2311.18402/) (2023)
- [Inaccurate Label Distribution Learning](https://mlsanity.com/paper/2302.13000/) (2023)
- [Language-Driven Visual Consensus for Zero-Shot Semantic Segmentation](https://mlsanity.com/paper/2403.08426/) (2024)
- [SVIP: Semantically Contextualized Visual Patches for Zero-Shot Learning](https://mlsanity.com/paper/2503.10252/) (2025)
- [On the Element-Wise Representation and Reasoning in Zero-Shot Image Recognition: A Systematic Survey](https://mlsanity.com/paper/2408.04879/) (2024)
- [Hierarchical Dynamic Masks for Visual Explanation of Neural Networks](https://mlsanity.com/paper/2301.04970/) (2023)
- [Sentinel-Guided Zero-Shot Learning: A Collaborative Paradigm without Real Data Exposure](https://mlsanity.com/paper/2403.09363/) (2024)
- [Uncertainty Guided Refinement for Fine-Grained Salient Object Detection](https://mlsanity.com/paper/2504.09666/) (2025)
- [Mutual Balancing in State-Object Components for Compositional Zero-Shot Learning](https://mlsanity.com/paper/2211.10647/) (2022)
- [An Integral Projection-based Semantic Autoencoder for Zero-Shot Learning](https://mlsanity.com/paper/2306.14628/) (2023)
- [Diverse and Tailored Image Generation for Zero-shot Multi-label Classification](https://mlsanity.com/paper/2404.03144/) (2024)
- [HPT++: Hierarchically Prompting Vision-Language Models with Multi-Granularity Knowledge Generation and Improved Structure Modeling](https://mlsanity.com/paper/2408.14812/) (2024)
- [MAC: A Benchmark for Multiple Attributes Compositional Zero-Shot Learning](https://mlsanity.com/paper/2406.12757/) (2024)
- [Prototype-Guided Curriculum Learning for Zero-Shot Learning](https://mlsanity.com/paper/2508.07771/) (2025)
- [Fine-Grained Zero-Shot Learning with Attribute-Centric Representations](https://mlsanity.com/paper/2512.12219/) (2025)

## Abstract (arXiv)

Zero-shot learning (ZSL) aims to learn models that can recognize unseen image semantics based on the training of data with seen semantics. Recent studies either leverage the global image features or mine discriminative local patch features to associate the extracted visual features to the semantic attributes. However, due to the lack of the necessary top-down guidance and semantic alignment for ensuring the model attending to the real attribute-correlation regions, these methods still encounter a significant semantic gap between the visual modality and the attribute modality, which makes their prediction on unseen semantics unreliable. To solve this problem, this paper establishes a novel transformer encoder-decoder model, called hybrid routing transformer (HRT). In HRT encoder, we embed an active attention, which is constructed by both the bottom-up and the top-down dynamic routing pathways to generate the attribute-aligned visual feature. While in HRT decoder, we use static routing to calculate the correlation among the attribute-aligned visual features, the corresponding attribute semantics, and the class attribute vectors to generate the final class label predictions. This design makes the presented transformer model a hybrid of 1) top-down and bottom-up attention pathways and 2) dynamic and static routing pathways. Comprehensive experiments on three widely-used benchmark datasets, namely CUB, SUN, and AWA2, are conducted. The obtained experimental results demonstrate the effectiveness of the proposed method.