# Goal-Oriented Gaze Estimation for Zero-Shot Learning

arXiv: https://arxiv.org/abs/2103.03433 · HTML: https://mlsanity.com/paper/2103.03433/

## Key facts

- Cited by 68 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 70 references that are in the map.
- First posted on arXiv 5 Mar 2021.

## Builds on (most cited first, 30 of 41)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Glow: Generative Flow with Invertible 1x1 Convolutions](https://mlsanity.com/paper/1807.03039/) (2018)
- [Learning to Compare: Relation Network for Few-Shot Learning](https://mlsanity.com/paper/1711.06025/) (2017)
- [Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Ugly](https://mlsanity.com/paper/1707.00600/) (2017)
- [Dynamic Few-Shot Visual Learning without Forgetting](https://mlsanity.com/paper/1804.09458/) (2018)
- [Feature Generating Networks for Zero-Shot Learning](https://mlsanity.com/paper/1712.00981/) (2017)
- [Zero-Shot Learning -- The Good, the Bad and the Ugly](https://mlsanity.com/paper/1703.04394/) (2017)
- [Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding](https://mlsanity.com/paper/1606.01847/) (2016)
- [Generalized Zero- and Few-Shot Learning via Aligned Variational Autoencoders](https://mlsanity.com/paper/1812.01784/) (2018)
- [Evaluation of Output Embeddings for Fine-Grained Image Classification](https://mlsanity.com/paper/1409.8403/) (2014)
- [Attribute Prototype Network for Zero-Shot Learning](https://mlsanity.com/paper/2008.08290/) (2020)
- [Learning a Deep Embedding Model for Zero-Shot Learning](https://mlsanity.com/paper/1611.05088/) (2016)
- [An Empirical Study and Analysis of Generalized Zero-Shot Learning for Object Recognition in the Wild](https://mlsanity.com/paper/1605.04253/) (2016)
- [Latent Embeddings for Zero-shot Classification](https://mlsanity.com/paper/1603.08895/) (2016)
- [Multi-modal Factorized Bilinear Pooling with Co-Attention Learning for Visual Question Answering](https://mlsanity.com/paper/1708.01471/) (2017)
- [Generalized Zero-Shot Learning via Synthesized Examples](https://mlsanity.com/paper/1712.03878/) (2017)
- [Multi-modal Cycle-consistent Generalized Zero-Shot Learning](https://mlsanity.com/paper/1808.00136/) (2018)
- [Predicting Gaze in Egocentric Video by Learning Task-dependent Attention Transition](https://mlsanity.com/paper/1803.09125/) (2018)
- [Domain-aware Visual Bias Eliminating for Generalized Zero-Shot Learning](https://mlsanity.com/paper/2003.13261/) (2020)
- [Zero-Shot Visual Recognition using Semantics-Preserving Adversarial Embedding Networks](https://mlsanity.com/paper/1712.01928/) (2017)
- [Episode-based Prototype Generating Network for Zero-Shot Learning](https://mlsanity.com/paper/1909.03360/) (2019)
- [Transferable Contrastive Network for Generalized Zero-Shot Learning](https://mlsanity.com/paper/1908.05832/) (2019)
- [Leveraging Seen and Unseen Semantic Relationships for Generative Zero-Shot Learning](https://mlsanity.com/paper/2007.09549/) (2020)
- [Cosine Normalization: Using Cosine Similarity Instead of Dot Product in Neural Networks](https://mlsanity.com/paper/1702.05870/) (2017)
- [Semantic-Guided Multi-Attention Localization for Zero-Shot Learning](https://mlsanity.com/paper/1903.00502/) (2019)
- [Generalized Zero-Shot Learning Via Over-Complete Distribution](https://mlsanity.com/paper/2004.00666/) (2020)
- [Mutual Context Network for Jointly Estimating Egocentric Gaze and Actions](https://mlsanity.com/paper/1901.01874/) (2019)
- [Invertible Zero-Shot Recognition Flows](https://mlsanity.com/paper/2007.04873/) (2020)
- [Transductive Unbiased Embedding for Zero-Shot Learning](https://mlsanity.com/paper/1803.11320/) (2018)

## Built on by (most cited first, 30 of 68)

- [TransZero: Attribute-guided Transformer for Zero-Shot Learning](https://mlsanity.com/paper/2112.01683/) (2021)
- [MSDN: Mutually Semantic Distillation Network for Zero-Shot Learning](https://mlsanity.com/paper/2203.03137/) (2022)
- [Decomposed Soft Prompt Guided Fusion Enhancing for Compositional Zero-Shot Learning](https://mlsanity.com/paper/2211.10681/) (2022)
- [DUET: Cross-modal Semantic Grounding for Contrastive Zero-shot Learning](https://mlsanity.com/paper/2207.01328/) (2022)
- [TransZero++: Cross Attribute-Guided Transformer for Zero-Shot Learning](https://mlsanity.com/paper/2112.08643/) (2021)
- [VGSE: Visually-Grounded Semantic Embeddings for Zero-Shot Learning](https://mlsanity.com/paper/2203.10444/) (2022)
- [Dual Progressive Prototype Network for Generalized Zero-Shot Learning](https://mlsanity.com/paper/2111.02073/) (2021)
- [Progressive Semantic-Visual Mutual Adaption for Generalized Zero-Shot Learning](https://mlsanity.com/paper/2303.15322/) (2023)
- [Zero-Shot Learning by Harnessing Adversarial Samples](https://mlsanity.com/paper/2308.00313/) (2023)
- [RadioTransformer: A Cascaded Global-Focal Transformer for Visual Attention-guided Disease Classification](https://mlsanity.com/paper/2202.11781/) (2022)
- [Attribute Prototype Network for Any-Shot Learning](https://mlsanity.com/paper/2204.01208/) (2022)
- [GaTector: A Unified Framework for Gaze Object Prediction](https://mlsanity.com/paper/2112.03549/) (2021)
- [DRPT: Disentangled and Recurrent Prompt Tuning for Compositional Zero-Shot Learning](https://mlsanity.com/paper/2305.01239/) (2023)
- [Progressive Semantic-Guided Vision Transformer for Zero-Shot Learning](https://mlsanity.com/paper/2404.07713/) (2024)
- [Learning from Observer Gaze:Zero-Shot Attention Prediction Oriented by Human-Object Interaction Recognition](https://mlsanity.com/paper/2405.09931/) (2024)
- [Visual-Augmented Dynamic Semantic Prototype for Generative Zero-Shot Learning](https://mlsanity.com/paper/2404.14808/) (2024)
- [Semantic-aligned Fusion Transformer for One-shot Object Detection](https://mlsanity.com/paper/2203.09093/) (2022)
- [Fine-Grained Zero-Shot Learning: Advances, Challenges, and Prospects](https://mlsanity.com/paper/2401.17766/) (2024)
- [CREST: Cross-modal Resonance through Evidential Deep Learning for Enhanced Zero-Shot Learning](https://mlsanity.com/paper/2404.09640/) (2024)
- [Disentangling Semantic-to-visual Confusion for Zero-shot Learning](https://mlsanity.com/paper/2106.08605/) (2021)
- [Learning Adversarial Semantic Embeddings for Zero-Shot Recognition in Open Worlds](https://mlsanity.com/paper/2307.03416/) (2023)
- [What Can Knowledge Bring to Machine Learning? -- A Survey of Low-shot Learning for Structured Data](https://mlsanity.com/paper/2106.06410/) (2021)
- [Where, What, Why: Towards Explainable Driver Attention Prediction](https://mlsanity.com/paper/2506.23088/) (2025)
- [Multi-Semantic Fusion Model for Generalized Zero-Shot Skeleton-Based Action Recognition](https://mlsanity.com/paper/2309.09592/) (2023)
- [Boosting Zero-shot Learning via Contrastive Optimization of Attribute Representations](https://mlsanity.com/paper/2207.03824/) (2022)
- [Rebalanced Zero-shot Learning](https://mlsanity.com/paper/2210.07031/) (2022)
- [Region Semantically Aligned Network for Zero-Shot Learning](https://mlsanity.com/paper/2110.07130/) (2021)
- [Deconstructed Generation-Based Zero-Shot Model](https://mlsanity.com/paper/2204.11280/) (2022)
- [SVIP: Semantically Contextualized Visual Patches for Zero-Shot Learning](https://mlsanity.com/paper/2503.10252/) (2025)
- [Implicit and Explicit Attention for Zero-Shot Learning](https://mlsanity.com/paper/2110.00860/) (2021)

## Abstract (arXiv)

Zero-shot learning (ZSL) aims to recognize novel classes by transferring semantic knowledge from seen classes to unseen classes. Since semantic knowledge is built on attributes shared between different classes, which are highly local, strong prior for localization of object attribute is beneficial for visual-semantic embedding. Interestingly, when recognizing unseen images, human would also automatically gaze at regions with certain semantic clue. Therefore, we introduce a novel goal-oriented gaze estimation module (GEM) to improve the discriminative attribute localization based on the class-level attributes for ZSL. We aim to predict the actual human gaze location to get the visual attention regions for recognizing a novel object guided by attribute description. Specifically, the task-dependent attention is learned with the goal-oriented GEM, and the global image features are simultaneously optimized with the regression of local attribute features. Experiments on three ZSL benchmarks, i.e., CUB, SUN and AWA2, show the superiority or competitiveness of our proposed method against the state-of-the-art ZSL methods. The ablation analysis on real gaze data CUB-VWSW also validates the benefits and accuracy of our gaze estimation module. This work implies the promising benefits of collecting human gaze dataset and automatic gaze estimation algorithms on high-level computer vision tasks. The code is available at https://github.com/osierboy/GEM-ZSL.