# HOIBlender: Blending Lightweight Detection with Vision-Language Priors for Efficient Human-Object Interaction Detection

arXiv: https://arxiv.org/abs/2609.23431 · HTML: https://mlsanity.com/paper/2609.23431/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 48 references that are in the map.
- First posted on arXiv 20 Sep 2026.

## Builds on (most cited first, 30 of 41)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR](https://mlsanity.com/paper/2201.12329/) (2022)
- [DETRs Beat YOLOs on Real-time Object Detection](https://mlsanity.com/paper/2304.08069/) (2023)
- [DN-DETR: Accelerate DETR Training by Introducing Query DeNoising](https://mlsanity.com/paper/2203.01305/) (2022)
- [Learning to Detect Human-Object Interactions](https://mlsanity.com/paper/1702.05448/) (2017)
- [Detecting and Recognizing Human-Object Interactions](https://mlsanity.com/paper/1704.07333/) (2017)
- [Visual Semantic Role Labeling](https://mlsanity.com/paper/1505.04474/) (2015)
- [HOTR: End-to-End Human-Object Interaction Detection with Transformers](https://mlsanity.com/paper/2104.13682/) (2021)
- [Learning Human-Object Interactions by Graph Parsing Neural Networks](https://mlsanity.com/paper/1808.07962/) (2018)
- [QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Information](https://mlsanity.com/paper/2103.05399/) (2021)
- [End-to-End Human Object Interaction Detection with HOI Transformer](https://mlsanity.com/paper/2103.04503/) (2021)
- [iCAN: Instance-Centric Attention Network for Human-Object Interaction Detection](https://mlsanity.com/paper/1808.10437/) (2018)
- [GEN-VLKT: Simplify Association and Enhance Interaction Understanding for HOI Detection](https://mlsanity.com/paper/2203.13954/) (2022)
- [Group DETR: Fast DETR Training with Group-Wise One-to-Many Assignment](https://mlsanity.com/paper/2207.13085/) (2022)
- [Reformulating HOI Detection as Adaptive Set Prediction](https://mlsanity.com/paper/2103.05983/) (2021)
- [Efficient Two-Stage Detection of Human-Object Interactions with a Novel Unary-Pairwise Transformer](https://mlsanity.com/paper/2112.01838/) (2021)
- [Pose-aware Multi-level Feature Network for Human Object Interaction Detection](https://mlsanity.com/paper/1909.08453/) (2019)
- [RF-DETR: Neural Architecture Search for Real-Time Detection Transformers](https://mlsanity.com/paper/2511.09554/) (2025)
- [HOICLIP: Efficient Knowledge Transfer for HOI Detection with Vision-Language Models](https://mlsanity.com/paper/2303.15786/) (2023)
- [Exploring Predicate Visual Context in Detecting Human-Object Interactions](https://mlsanity.com/paper/2308.06202/) (2023)

## Built on by (most cited first, 1 of 1)

- [Harnessing Multimodal Large Language Models for Training-Free Human-Object Interaction Detection](https://mlsanity.com/paper/2610.06394/) (2026)

## Abstract (arXiv)

Human-object interaction (HOI) detection requires grounding an interacting human-object pair and recognizing the verb that links them, often under severe long-tail supervision. Recent methods improve accuracy with stronger detectors and vision-language priors, but many still stack heavy transformer encoders, intricate denoising schedules, or post-hoc semantic calibration on top of the detector. We present HOIBlender, an efficient HOI detector named after its core design principle: blending detector-grounded visual tokens, spatial subject-object reasoning, and BLIP-2 semantic priors inside one lightweight decoding pipeline. HOIBlender builds on an RF-DETR/LW-DETR-style foundation with a DINOv2 backbone and selects top-K image-conditioned tokens directly from the multi-scale projector as subject and object candidates, removing the dedicated encoder stage retained by prior HOI methods. A dual-stage decoder first stabilizes human-object geometry and then performs verb and HOI classification through progressive BLIP-2 prior fusion, with classifier weights initialized from BLIP-2 text embeddings for long-tail categories. Grouped-query training further enriches optimization without increasing inference cost. Across three model scales (Nano, Small, 2XL), HOIBlender consistently outperforms SOV-STG-VLA and Hybrid-SOV-VLA on HICO-DET, reaching 44.49 Default Full mAP in only 9 training epochs while maintaining competitive latency and parameter budgets. These results show that lightweight detection, structured spatial-semantic decoding, and deeply integrated vision-language priors can be blended into a single efficient HOI pipeline.