# Interaction-aware Joint Attention Estimation Using People Attributes

arXiv: https://arxiv.org/abs/2308.05382 · HTML: https://mlsanity.com/paper/2308.05382/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 60 references that are in the map.
- First posted on arXiv 10 Aug 2023.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Objects as Points](https://mlsanity.com/paper/1904.07850/) (2019)
- [Action-Conditioned 3D Human Motion Synthesis with Transformer VAE](https://mlsanity.com/paper/2104.05670/) (2021)
- [Gaze360: Physically Unconstrained Gaze Estimation in the Wild](https://mlsanity.com/paper/1910.10088/) (2019)
- [TokenPose: Learning Keypoint Tokens for Human Pose Estimation](https://mlsanity.com/paper/2104.03516/) (2021)
- [Hierarchical Deep Temporal Models for Group Activity Recognition](https://mlsanity.com/paper/1607.02643/) (2016)
- [Actor-Transformers for Group Activity Recognition](https://mlsanity.com/paper/2003.12737/) (2020)
- [Learning Actor Relation Graphs for Group Activity Recognition](https://mlsanity.com/paper/1904.10117/) (2019)
- [Detecting Attended Visual Targets in Video](https://mlsanity.com/paper/2003.02501/) (2020)
- [Glance and Gaze: Inferring Action-aware Points for One-Stage Human-Object Interaction Detection](https://mlsanity.com/paper/2104.05269/) (2021)
- [Joint Learning of Saliency Detection and Weakly Supervised Semantic Segmentation](https://mlsanity.com/paper/1909.04161/) (2019)
- [GroupFormer: Group Activity Recognition with Clustered Spatial-Temporal Transformer](https://mlsanity.com/paper/2108.12630/) (2021)
- [Connecting Gaze, Scene, and Attention: Generalized Attention Estimation via Joint Modeling of Gaze and Scene Saliency](https://mlsanity.com/paper/1807.10437/) (2018)
- [Deep Pictorial Gaze Estimation](https://mlsanity.com/paper/1807.10002/) (2018)
- [WHENet: Real-time Fine-Grained Estimation for Wide Range Head Pose](https://mlsanity.com/paper/2005.10353/) (2020)
- [End-to-End Human-Gaze-Target Detection with Transformers](https://mlsanity.com/paper/2203.10433/) (2022)
- [Joint Learning of Social Groups, Individuals Action and Sub-group Activities in Videos](https://mlsanity.com/paper/2007.02632/) (2020)
- [Believe It or Not, We Know What You Are Looking at!](https://mlsanity.com/paper/1907.02364/) (2019)
- [Encouraging LSTMs to Anticipate Actions Very Early](https://mlsanity.com/paper/1703.07023/) (2017)
- [Exploiting saliency for object segmentation from image level labels](https://mlsanity.com/paper/1701.08261/) (2017)
- [Weakly-Supervised Physically Unconstrained Gaze Estimation](https://mlsanity.com/paper/2105.09803/) (2021)
- [Dual-AI: Dual-path Actor Interaction Learning for Group Activity Recognition](https://mlsanity.com/paper/2204.02148/) (2022)
- [A Differential Approach for Gaze Estimation](https://mlsanity.com/paper/1904.09459/) (2019)
- [Deep Saliency with Encoded Low level Distance Map and High Level Features](https://mlsanity.com/paper/1604.05495/) (2016)
- [GAN-based Reactive Motion Synthesis with Class-aware Discriminators for Human-human Interaction](https://mlsanity.com/paper/2110.00380/) (2021)
- [Pose for Action - Action for Pose](https://mlsanity.com/paper/1603.04037/) (2016)
- [Attention Flow: End-to-End Joint Attention Estimation](https://mlsanity.com/paper/2001.03960/) (2020)
- [GROWL: Group Detection With Link Prediction](https://mlsanity.com/paper/2111.04397/) (2021)

## Built on by (most cited first, 6 of 6)

- [Gaze-LLE: Gaze Target Estimation via Large-Scale Learned Encoders](https://mlsanity.com/paper/2412.09586/) (2024)
- [Modeling Multimodal Social Interactions: New Challenges and Baselines with Densely Aligned Representations](https://mlsanity.com/paper/2403.02090/) (2024)
- [Learning Group Activity Features Through Person Attribute Prediction](https://mlsanity.com/paper/2403.02753/) (2024)
- [Semi-Supervised Unconstrained Head Pose Estimation in the Wild](https://mlsanity.com/paper/2404.02544/) (2024)
- [End-to-End Shared Attention Estimation via Group Detection with Feedback Refinement](https://mlsanity.com/paper/2604.01714/) (2026)
- [From Gaze to Meaning: A Training-Free AI Agent for Unified Grounding and Explanation](https://mlsanity.com/paper/2609.06208/) (2026)

## Abstract (arXiv)

This paper proposes joint attention estimation in a single image. Different from related work in which only the gaze-related attributes of people are independently employed, (I) their locations and actions are also employed as contextual cues for weighting their attributes, and (ii) interactions among all of these attributes are explicitly modeled in our method. For the interaction modeling, we propose a novel Transformer-based attention network to encode joint attention as low-dimensional features. We introduce a specialized MLP head with positional embedding to the Transformer so that it predicts pixelwise confidence of joint attention for generating the confidence heatmap. This pixelwise prediction improves the heatmap accuracy by avoiding the ill-posed problem in which the high-dimensional heatmap is predicted from the low-dimensional features. The estimated joint attention is further improved by being integrated with general image-based attention estimation. Our method outperforms SOTA methods quantitatively in comparative experiments. Code: https://anonymous.4open.science/r/anonymized_codes-ECA4.