# Weakly Supervised Attended Object Detection Using Gaze Data as Annotations

arXiv: https://arxiv.org/abs/2204.07090 · HTML: https://mlsanity.com/paper/2204.07090/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 37 references that are in the map.
- First posted on arXiv 14 Apr 2022.

## Builds on (most cited first, 19 of 19)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [What's the Point: Semantic Segmentation with Point Supervision](https://mlsanity.com/paper/1506.02106/) (2015)
- [The Secrets of Salient Object Segmentation](https://mlsanity.com/paper/1406.2807/) (2014)
- [Simultaneous Detection and Segmentation](https://mlsanity.com/paper/1407.1808/) (2014)
- [Rolling-Unrolling LSTMs for Action Anticipation from First-Person Video](https://mlsanity.com/paper/2005.02190/) (2020)
- [Pointly-Supervised Instance Segmentation](https://mlsanity.com/paper/2104.06404/) (2021)
- [A Dataset and Baselines for Visual Question Answering on Art](https://mlsanity.com/paper/2008.12520/) (2020)
- [A Self-Training Approach for Point-Supervised Object Detection and Counting in Crowds](https://mlsanity.com/paper/2007.12831/) (2020)
- [PseudoEdgeNet: Nuclei Segmentation only with Point Annotations](https://mlsanity.com/paper/1906.02924/) (2019)
- [EGO-CH: Dataset and Fundamental Tasks for Visitors BehavioralUnderstanding using Egocentric Vision](https://mlsanity.com/paper/2002.00899/) (2020)

## Built on by (most cited first, 5 of 5)

- [Object-aware Gaze Target Detection](https://mlsanity.com/paper/2307.09662/) (2023)
- [Eyes on Target: Gaze-Aware Object Detection in Egocentric Video](https://mlsanity.com/paper/2511.01237/) (2025)
- [AL-GTD: Deep Active Learning for Gaze Target Detection](https://mlsanity.com/paper/2409.18561/) (2024)
- [Data Augmentation via Latent Diffusion Models for Detecting Smell-Related Objects in Historical Artworks](https://mlsanity.com/paper/2509.14755/) (2025)
- [From Ambiguity to Action: A POMDP Perspective on Partial Multi-Label Ambiguity and Its Horizon-One Resolution](https://mlsanity.com/paper/2602.04255/) (2026)

## Abstract (arXiv)

We consider the problem of detecting and recognizing the objects observed by visitors (i.e., attended objects) in cultural sites from egocentric vision. A standard approach to the problem involves detecting all objects and selecting the one which best overlaps with the gaze of the visitor, measured through a gaze tracker. Since labeling large amounts of data to train a standard object detector is expensive in terms of costs and time, we propose a weakly supervised version of the task which leans only on gaze data and a frame-level label indicating the class of the attended object. To study the problem, we present a new dataset composed of egocentric videos and gaze coordinates of subjects visiting a museum. We hence compare three different baselines for weakly supervised attended object detection on the collected data. Results show that the considered approaches achieve satisfactory performance in a weakly supervised manner, which allows for significant time savings with respect to a fully supervised detector based on Faster R-CNN. To encourage research on the topic, we publicly release the code and the dataset at the following url: https://iplab.dmi.unict.it/WS_OBJ_DET/