# Geometric Visual Fusion Graph Neural Networks for Multi-Person Human-Object Interaction Recognition in Videos

arXiv: https://arxiv.org/abs/2506.03440 · HTML: https://mlsanity.com/paper/2506.03440/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 59 of its 82 references that are in the map.
- First posted on arXiv 3 Jun 2025.

## Builds on (most cited first, 30 of 59)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [How Attentive are Graph Attention Networks?](https://mlsanity.com/paper/2105.14491/) (2021)
- [Network In Network](https://mlsanity.com/paper/1312.4400/) (2013)
- [Rescaling Egocentric Vision](https://mlsanity.com/paper/2006.13256/) (2020)
- [VectorNet: Encoding HD Maps and Agent Dynamics from Vectorized Representation](https://mlsanity.com/paper/2005.04259/) (2020)
- [Temporal Convolutional Networks for Action Segmentation and Detection](https://mlsanity.com/paper/1611.05267/) (2016)
- [Learning Lane Graph Representations for Motion Forecasting](https://mlsanity.com/paper/2007.13732/) (2020)
- [MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation](https://mlsanity.com/paper/1903.01945/) (2019)
- [HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interaction](https://mlsanity.com/paper/2203.01577/) (2022)
- [BEHAVE: Dataset and Method for Tracking Human Object Interactions](https://mlsanity.com/paper/2204.06950/) (2022)
- [Deep Learning-Based Human Pose Estimation: A Survey](https://mlsanity.com/paper/2012.13392/) (2020)
- [Populating 3D Scenes by Learning Human-Scene Interaction](https://mlsanity.com/paper/2012.11581/) (2020)
- [Bottom-up Object Detection by Grouping Extreme and Center Points](https://mlsanity.com/paper/1901.08043/) (2019)
- [Something-Else: Compositional Action Recognition with Spatial-Temporal Interaction Networks](https://mlsanity.com/paper/1912.09930/) (2019)
- [Pose-aware Multi-level Feature Network for Human Object Interaction Detection](https://mlsanity.com/paper/1909.08453/) (2019)
- [VPN: Learning Video-Pose Embedding for Activities of Daily Living](https://mlsanity.com/paper/2007.03056/) (2020)
- [DECO: Dense Estimation of 3D Human-Scene Contact In The Wild](https://mlsanity.com/paper/2309.15273/) (2023)
- [LEMON: Learning 3D Human-Object Interaction Relation from 2D Images](https://mlsanity.com/paper/2312.08963/) (2023)
- [Exploring Structure-aware Transformer over Interaction Proposals for Human-Object Interaction Detection](https://mlsanity.com/paper/2206.06291/) (2022)
- [From Lifestyle Vlogs to Everyday Interactions](https://mlsanity.com/paper/1712.02310/) (2017)
- [ViPLO: Vision Transformer based Pose-Conditioned Self-Loop Graph for Human-Object Interaction Detection](https://mlsanity.com/paper/2304.08114/) (2023)
- [Relational Context Learning for Human-Object Interaction Detection](https://mlsanity.com/paper/2304.04997/) (2023)
- [Mining Cross-Person Cues for Body-Part Interactiveness Learning in HOI Detection](https://mlsanity.com/paper/2207.14192/) (2022)
- [Glimpse Clouds: Human Activity Recognition from Unstructured Feature Points](https://mlsanity.com/paper/1802.07898/) (2018)

## Built on by (most cited first, 3 of 3)

- [Egocentric Human-Object Interaction Detection: A New Benchmark and Method](https://mlsanity.com/paper/2506.14189/) (2025)
- [ART: Adaptive Relational Transformer for Pedestrian Trajectory Prediction with Temporal-Aware Relations](https://mlsanity.com/paper/2604.03649/) (2026)
- [HSTGFormer: Hyper Spatial-Temporal Graph Transformer for 3D Human Pose Estimation](https://mlsanity.com/paper/2608.12187/) (2026)

## Abstract (arXiv)

Human-Object Interaction (HOI) recognition in videos requires understanding both visual patterns and geometric relationships as they evolve over time. Visual and geometric features offer complementary strengths. Visual features capture appearance context, while geometric features provide structural patterns. Effectively fusing these multimodal features without compromising their unique characteristics remains challenging. We observe that establishing robust, entity-specific representations before modeling interactions helps preserve the strengths of each modality. Therefore, we hypothesize that a bottom-up approach is crucial for effective multimodal fusion. Following this insight, we propose the Geometric Visual Fusion Graph Neural Network (GeoVis-GNN), which uses dual-attention feature fusion combined with interdependent entity graph learning. It progressively builds from entity-specific representations toward high-level interaction understanding. To advance HOI recognition to real-world scenarios, we introduce the Concurrent Partial Interaction Dataset (MPHOI-120). It captures dynamic multi-person interactions involving concurrent actions and partial engagement. This dataset helps address challenges like complex human-object dynamics and mutual occlusions. Extensive experiments demonstrate the effectiveness of our method across various HOI scenarios. These scenarios include two-person interactions, single-person activities, bimanual manipulations, and complex concurrent partial interactions. Our method achieves state-of-the-art performance.