# Gaze Embeddings for Zero-Shot Image Classification

arXiv: https://arxiv.org/abs/1611.09309 · HTML: https://mlsanity.com/paper/1611.09309/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Nov 2016.

## Built on by (most cited first, 25 of 25)

- [PointCLIP: Point Cloud Understanding by CLIP](https://mlsanity.com/paper/2112.02413/) (2021)
- [Goal-Oriented Gaze Estimation for Zero-Shot Learning](https://mlsanity.com/paper/2103.03433/) (2021)
- [Follow My Eye: Using Gaze to Supervise Computer-Aided Diagnosis](https://mlsanity.com/paper/2204.02976/) (2022)
- [Automatic Gaze Analysis: A Survey of Deep Learning based Approaches](https://mlsanity.com/paper/2108.05479/) (2021)
- [Human Attention in Fine-grained Classification](https://mlsanity.com/paper/2111.01628/) (2021)
- [Multimodal Integration of Human-Like Attention in Visual Question Answering](https://mlsanity.com/paper/2109.13139/) (2021)
- [Fine-Grained Zero-Shot Learning with DNA as Side Information](https://mlsanity.com/paper/2109.14133/) (2021)
- [Synthesizing Human Gaze Feedback for Improved NLP Performance](https://mlsanity.com/paper/2302.05721/) (2023)
- [Rectify ViT Shortcut Learning by Visual Saliency](https://mlsanity.com/paper/2206.08567/) (2022)
- [Disentangling Semantic-to-visual Confusion for Zero-shot Learning](https://mlsanity.com/paper/2106.08605/) (2021)
- [MTGLS: Multi-Task Gaze Estimation with Limited Supervision](https://mlsanity.com/paper/2110.12100/) (2021)
- [EyeTrans: Merging Human and Machine Attention for Neural Code Summarization](https://mlsanity.com/paper/2402.14096/) (2024)
- [Gaze-Guided Class Activation Mapping: Leveraging Human Attention for Network Attention in Chest X-rays Classification](https://mlsanity.com/paper/2202.07107/) (2022)
- [Multi-task UNet: Jointly Boosting Saliency Prediction and Disease Classification on Chest X-ray Images](https://mlsanity.com/paper/2202.07118/) (2022)
- [Neglected Free Lunch -- Learning Image Classifiers Using Annotation Byproducts](https://mlsanity.com/paper/2303.17595/) (2023)
- [Learning Better Contrastive View from Radiologist's Gaze](https://mlsanity.com/paper/2305.08826/) (2023)
- [Gaze-Guided Learning: Avoiding Shortcut Bias in Visual Classification](https://mlsanity.com/paper/2504.05583/) (2025)
- [Where to look at the movies : Analyzing visual attention to understand movie editing](https://mlsanity.com/paper/2102.13378/) (2021)
- [Eyes on Target: Gaze-Aware Object Detection in Egocentric Video](https://mlsanity.com/paper/2511.01237/) (2025)
- [Zero-shot Learning with Deep Neural Networks for Object Recognition](https://mlsanity.com/paper/2102.03137/) (2021)
- [TEPI: Taxonomy-aware Embedding and Pseudo-Imaging for Scarcely-labeled Zero-shot Genome Classification](https://mlsanity.com/paper/2401.13219/) (2024)
- [IRIS: Intent Resolution via Inference-time Saccades for Open-Ended VQA in Large Vision-Language Models](https://mlsanity.com/paper/2602.16138/) (2026)
- [A Simple Approach for Zero-Shot Learning based on Triplet Distribution Embeddings](https://mlsanity.com/paper/2103.15939/) (2021)
- [RAZE: Region Guided Self-Supervised Gaze Representation Learning](https://mlsanity.com/paper/2208.02485/) (2022)
- [Can Webcam Gaze Constrain Mesa-Objectives in Driving Models? An Instrument Precision Analysis](https://mlsanity.com/paper/2608.08947/) (2026)

## Abstract (arXiv)

Zero-shot image classification using auxiliary information, such as attributes describing discriminative object properties, requires time-consuming annotation by domain experts. We instead propose a method that relies on human gaze as auxiliary information, exploiting that even non-expert users have a natural ability to judge class membership. We present a data collection paradigm that involves a discrimination task to increase the information content obtained from gaze data. Our method extracts discriminative descriptors from the data and learns a compatibility function between image and gaze using three novel gaze embeddings: Gaze Histograms (GH), Gaze Features with Grid (GFG) and Gaze Features with Sequence (GFS). We introduce two new gaze-annotated datasets for fine-grained image classification and show that human gaze data is indeed class discriminative, provides a competitive alternative to expert-annotated attributes, and outperforms other baselines for zero-shot image classification.