# FineHOI: Part-Aware Dense Representations for Zero-Shot Human-Object Interaction Detection

arXiv: https://arxiv.org/abs/2609.05959 · HTML: https://mlsanity.com/paper/2609.05959/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 38 references that are in the map.
- First posted on arXiv 5 Sep 2026.

## Builds on (most cited first, 26 of 26)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Learning to Detect Human-Object Interactions](https://mlsanity.com/paper/1702.05448/) (2017)
- [Visual Semantic Role Labeling](https://mlsanity.com/paper/1505.04474/) (2015)
- [Image Captioning: Transforming Objects into Words](https://mlsanity.com/paper/1906.05963/) (2019)
- [QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Information](https://mlsanity.com/paper/2103.05399/) (2021)
- [Visual Compositional Learning for Human-Object Interaction Detection](https://mlsanity.com/paper/2007.12407/) (2020)
- [DRG: Dual Relation Graph for Human-Object Interaction Detection](https://mlsanity.com/paper/2008.11714/) (2020)
- [End-to-end Generative Pretraining for Multimodal Video Captioning](https://mlsanity.com/paper/2201.08264/) (2022)
- [VSGNet: Spatial Attention Network for Detecting Human Object Interactions Using Graph Convolutions](https://mlsanity.com/paper/2003.05541/) (2020)
- [Mining the Benefits of Two-stage and One-stage HOI Detection](https://mlsanity.com/paper/2108.05077/) (2021)
- [Spatially Conditioned Graphs for Detecting Human-Object Interactions](https://mlsanity.com/paper/2012.06060/) (2020)
- [Detecting Human-Object Interaction via Fabricated Compositional Learning](https://mlsanity.com/paper/2103.08214/) (2021)
- [Exploring Predicate Visual Context in Detecting Human-Object Interactions](https://mlsanity.com/paper/2308.06202/) (2023)
- [Detecting Human-Object Interactions via Functional Generalization](https://mlsanity.com/paper/1904.03181/) (2019)
- [Efficient Adaptive Human-Object Interaction Detection with Concept-guided Memory](https://mlsanity.com/paper/2309.03696/) (2023)
- [Asynchronous Large Language Model Enhanced Planner for Autonomous Driving](https://mlsanity.com/paper/2406.14556/) (2024)
- [EZ-HOI: VLM Adaptation via Guided Prompt Learning for Zero-Shot HOI Detection](https://mlsanity.com/paper/2410.23904/) (2024)
- [Exploring Conditional Multi-Modal Prompts for Zero-shot HOI Detection](https://mlsanity.com/paper/2408.02484/) (2024)
- [Iwin: Human-Object Interaction Detection via Transformer with Irregular Windows](https://mlsanity.com/paper/2203.10537/) (2022)
- [Unseen No More: Unlocking the Potential of CLIP for Generative Zero-shot HOI Detection](https://mlsanity.com/paper/2408.05974/) (2024)
- [AlignedCut: Visual Concepts Discovery on Brain-Guided Universal Feature Space](https://mlsanity.com/paper/2406.18344/) (2024)
- [Visual Diversity and Region-aware Prompt Learning for Zero-shot HOI Detection](https://mlsanity.com/paper/2510.25094/) (2025)

## Abstract (arXiv)

Human-Object Interaction (HOI) detection aims to localize humans and objects in images and classify their interactions. Zero-shot HOI focuses on recognizing interactions that are not observed during training, requiring models to generalize beyond seen verb-object compositions. Recent approaches leverage Vision-Language Models (VLMs), benefiting from rich semantic representations. However, they often rely on global or detector-centric features that compress interaction cues and hinder fine-grained spatial reasoning. To overcome this limitation, we propose FineHOI, a zero-shot HOI framework that explicitly models interactions from dense patch-level features. Our approach is motivated by the observation that human-object interactions are defined by localized spatial relationships, which are not preserved by global and detector-centric representations. To this end, we introduce an Adaptive Part-Level Attention module that decomposes humans and objects into semantically coherent parts via unsupervised clustering, and re-weights them based on their interaction relevance. These representations are then integrated through a Region-Aware Interaction Transformer that integrates part-aware and global features and produces the final HOI embedding. Extensive experiments demonstrate that FineHOI consistently outperforms existing zero-shot HOI methods, achieving particularly strong gains on unseen interactions. Code is available at https://github.com/francescotonini/fine-hoi.