# Bilateral Collaboration with Large Vision-Language Models for Open Vocabulary Human-Object Interaction Detection

arXiv: https://arxiv.org/abs/2507.06510 · HTML: https://mlsanity.com/paper/2507.06510/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Driving VLMs and open-vocabulary detection, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 4 of its 85 references that are in the map.
- First posted on arXiv 9 Jul 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Driving VLMs and open-vocabulary detection](https://mlsanity.com/mllm/driving-vlms-and-open-vocabulary-detection/)

## Builds on (most cited first, 4 of 4)

- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [Visual Semantic Role Labeling](https://mlsanity.com/paper/1505.04474/) (2015)
- [CL-HOI: Cross-Level Human-Object Interaction Distillation from Vision Large Language Models](https://mlsanity.com/paper/2410.15657/) (2024)

## Built on by (most cited first, 6 of 6)

- [Graph is a Substrate Across Data Modalities](https://mlsanity.com/paper/2601.22384/) (2026)
- [What if Agents Could Imagine? Reinforcing Open-Vocabulary HOI Comprehension through Generation](https://mlsanity.com/paper/2602.11499/) (2026)
- [CrossHOI-Bench: A Unified Benchmark for HOI Evaluation across Vision-Language Models and HOI-Specific Methods](https://mlsanity.com/paper/2508.18753/) (2025)
- [Streamlined Open-Vocabulary Human-Object Interaction Detection](https://mlsanity.com/paper/2603.27500/) (2026)
- [Zero-shot HOI Detection with MLLM-based Detector-agnostic Interaction Recognition](https://mlsanity.com/paper/2602.15124/) (2026)
- [Generative Human-Object Interaction Detection via Differentiable Cognitive Steering of Multi-modal LLMs](https://mlsanity.com/paper/2512.17640/) (2025)

## Abstract (arXiv)

Open vocabulary Human-Object Interaction (HOI) detection is a challenging task that detects all <human, verb, object> triplets of interest in an image, even those that are not pre-defined in the training set. Existing approaches typically rely on output features generated by large Vision-Language Models (VLMs) to enhance the generalization ability of interaction representations. However, the visual features produced by VLMs are holistic and coarse-grained, which contradicts the nature of detection tasks. To address this issue, we propose a novel Bilateral Collaboration framework for open vocabulary HOI detection (BC-HOI). This framework includes an Attention Bias Guidance (ABG) component, which guides the VLM to produce fine-grained instance-level interaction features according to the attention bias provided by the HOI detector. It also includes a Large Language Model (LLM)-based Supervision Guidance (LSG) component, which provides fine-grained token-level supervision for the HOI detector by the LLM component of the VLM. LSG enhances the ability of ABG to generate high-quality attention bias. We conduct extensive experiments on two popular benchmarks: HICO-DET and V-COCO, consistently achieving superior performance in the open vocabulary and closed settings. The code will be released in Github.