# An analysis of HOI: using a training-free method with multimodal visual foundation models when only the test set is available, without the training set

arXiv: https://arxiv.org/abs/2408.05772 · HTML: https://mlsanity.com/paper/2408.05772/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 22 references that are in the map.
- First posted on arXiv 11 Aug 2024.

## Builds on (most cited first, 17 of 17)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Learning to Detect Human-Object Interactions](https://mlsanity.com/paper/1702.05448/) (2017)
- [HOTR: End-to-End Human-Object Interaction Detection with Transformers](https://mlsanity.com/paper/2104.13682/) (2021)
- [QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Information](https://mlsanity.com/paper/2103.05399/) (2021)
- [GEN-VLKT: Simplify Association and Enhance Interaction Understanding for HOI Detection](https://mlsanity.com/paper/2203.13954/) (2022)
- [VSGNet: Spatial Attention Network for Detecting Human Object Interactions Using Graph Convolutions](https://mlsanity.com/paper/2003.05541/) (2020)
- [Mining the Benefits of Two-stage and One-stage HOI Detection](https://mlsanity.com/paper/2108.05077/) (2021)
- [Detailed 2D-3D Joint Representation for Human-Object Interaction](https://mlsanity.com/paper/2004.08154/) (2020)
- [HOI Analysis: Integrating and Decomposing Human-Object Interaction](https://mlsanity.com/paper/2010.16219/) (2020)
- [Detecting Human-Object Interactions with Action Co-occurrence Priors](https://mlsanity.com/paper/2007.08728/) (2020)
- [RLIP: Relational Language-Image Pre-training for Human-Object Interaction Detection](https://mlsanity.com/paper/2209.01814/) (2022)
- [Interactiveness Field in Human-Object Interactions](https://mlsanity.com/paper/2204.07718/) (2022)
- [Mining Cross-Person Cues for Body-Part Interactiveness Learning in HOI Detection](https://mlsanity.com/paper/2207.14192/) (2022)
- [End-to-End Zero-Shot HOI Detection via Vision and Language Knowledge Distillation](https://mlsanity.com/paper/2204.03541/) (2022)
- [Weakly-supervised HOI Detection via Prior-guided Bi-level Representation Learning](https://mlsanity.com/paper/2303.01313/) (2023)

## Built on by (most cited first, 1 of 1)

- [Harnessing Multimodal Large Language Models for Training-Free Human-Object Interaction Detection](https://mlsanity.com/paper/2610.06394/) (2026)

## Abstract (arXiv)

Human-Object Interaction (HOI) aims to identify the pairs of humans and objects in images and to recognize their relationships, ultimately forming langle human, object, verb rangle triplets. Under default settings, HOI performance is nearly saturated, with many studies focusing on long-tail distribution and zero-shot/few-shot scenarios. Let us consider an intriguing problem:``What if there is only test dataset without training dataset, using multimodal visual foundation model in a training-free manner? '' This study uses two experimental settings: grounding truth and random arbitrary combinations. We get some interesting conclusion and find that the open vocabulary capabilities of the multimodal visual foundation model are not yet fully realized. Additionally, replacing the feature extraction with grounding DINO further confirms these findings.