# Efficient Adaptive Human-Object Interaction Detection with Concept-guided Memory

arXiv: https://arxiv.org/abs/2309.03696 · HTML: https://mlsanity.com/paper/2309.03696/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 53 references that are in the map.
- First posted on arXiv 7 Sep 2023.

## Builds on (most cited first, 30 of 45)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://mlsanity.com/paper/2102.03334/) (2021)
- [Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks](https://mlsanity.com/paper/2004.06165/) (2020)
- [CLIP-Adapter: Better Vision-Language Models with Feature Adapters](https://mlsanity.com/paper/2110.04544/) (2021)
- [AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition](https://mlsanity.com/paper/2205.13535/) (2022)
- [The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale](https://mlsanity.com/paper/1811.00982/) (2018)
- [Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling](https://mlsanity.com/paper/2111.03930/) (2021)
- [VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks](https://mlsanity.com/paper/2112.06825/) (2021)
- [Learning to Detect Human-Object Interactions](https://mlsanity.com/paper/1702.05448/) (2017)
- [Understanding the Difficulty of Training Transformers](https://mlsanity.com/paper/2004.08249/) (2020)
- [Detecting and Recognizing Human-Object Interactions](https://mlsanity.com/paper/1704.07333/) (2017)
- [Visual Semantic Role Labeling](https://mlsanity.com/paper/1505.04474/) (2015)
- [HOTR: End-to-End Human-Object Interaction Detection with Transformers](https://mlsanity.com/paper/2104.13682/) (2021)
- [Learning Human-Object Interactions by Graph Parsing Neural Networks](https://mlsanity.com/paper/1808.07962/) (2018)
- [QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Information](https://mlsanity.com/paper/2103.05399/) (2021)
- [PPDM: Parallel Point Detection and Matching for Real-time Human-Object Interaction Detection](https://mlsanity.com/paper/1912.12898/) (2019)
- [End-to-End Human Object Interaction Detection with HOI Transformer](https://mlsanity.com/paper/2103.04503/) (2021)
- [Visual Compositional Learning for Human-Object Interaction Detection](https://mlsanity.com/paper/2007.12407/) (2020)
- [iCAN: Instance-Centric Attention Network for Human-Object Interaction Detection](https://mlsanity.com/paper/1808.10437/) (2018)
- [GEN-VLKT: Simplify Association and Enhance Interaction Understanding for HOI Detection](https://mlsanity.com/paper/2203.13954/) (2022)
- [DRG: Dual Relation Graph for Human-Object Interaction Detection](https://mlsanity.com/paper/2008.11714/) (2020)

## Built on by (most cited first, 30 of 38)

- [EZ-HOI: VLM Adaptation via Guided Prompt Learning for Zero-Shot HOI Detection](https://mlsanity.com/paper/2410.23904/) (2024)
- [Exploring Conditional Multi-Modal Prompts for Zero-shot HOI Detection](https://mlsanity.com/paper/2408.02484/) (2024)
- [HOIGen-1M: A Large-scale Dataset for Human-Object Interaction Video Generation](https://mlsanity.com/paper/2503.23715/) (2025)
- [Exploring the Potential of Large Foundation Models for Open-Vocabulary HOI Detection](https://mlsanity.com/paper/2404.06194/) (2024)
- [Human-Object Interaction Detection Collaborated with Large Relation-driven Diffusion Models](https://mlsanity.com/paper/2410.20155/) (2024)
- [Locality-Aware Zero-Shot Human-Object Interaction Detection](https://mlsanity.com/paper/2505.19503/) (2025)
- [HOLa: Zero-Shot HOI Detection with Low-Rank Decomposed VLM Feature Adaptation](https://mlsanity.com/paper/2507.15542/) (2025)
- [Open-Vocabulary HOI Detection with Interaction-aware Prompt and Concept Calibration](https://mlsanity.com/paper/2508.03207/) (2025)
- [Hierarchical Event Memory for Accurate and Low-latency Online Video Temporal Grounding](https://mlsanity.com/paper/2508.04546/) (2025)
- [CrossHOI-Bench: A Unified Benchmark for HOI Evaluation across Vision-Language Models and HOI-Specific Methods](https://mlsanity.com/paper/2508.18753/) (2025)
- [Learning Human-Object Interaction as Groups](https://mlsanity.com/paper/2510.18357/) (2025)
- [Towards Zero-shot Human-Object Interaction Detection via Vision-Language Integration](https://mlsanity.com/paper/2403.07246/) (2024)
- [VHOI: Controllable Video Generation of Human-Object Interactions from Sparse Trajectories via Motion Densification](https://mlsanity.com/paper/2512.09646/) (2025)
- [Enhancing HOI Detection with Contextual Cues from Large Vision-Language Models](https://mlsanity.com/paper/2311.16475/) (2023)
- [SGC-Net: Stratified Granular Comparison Network for Open-Vocabulary HOI Detection](https://mlsanity.com/paper/2503.00414/) (2025)
- [Egocentric Human-Object Interaction Detection: A New Benchmark and Method](https://mlsanity.com/paper/2506.14189/) (2025)
- [TRKT: Weakly Supervised Dynamic Scene Graph Generation with Temporal-enhanced Relation-aware Knowledge Transferring](https://mlsanity.com/paper/2508.04943/) (2025)
- [No More Sibling Rivalry: Debiasing Human-Object Interaction Detection](https://mlsanity.com/paper/2509.00760/) (2025)
- [InteractMove: Text-Controlled Human-Object Interaction Generation in 3D Scenes with Movable Objects](https://mlsanity.com/paper/2509.23612/) (2025)
- [Visual Diversity and Region-aware Prompt Learning for Zero-shot HOI Detection](https://mlsanity.com/paper/2510.25094/) (2025)
- [Orchestrating the Symphony of Prompt Distribution Learning for Human-Object Interaction Detection](https://mlsanity.com/paper/2412.08506/) (2024)
- [Dynamic Scoring with Enhanced Semantics for Training-Free Human-Object Interaction Detection](https://mlsanity.com/paper/2507.17456/) (2025)
- [DQEN: Dual Query Enhancement Network for DETR-based HOI Detection](https://mlsanity.com/paper/2508.18896/) (2025)
- [Mitigating Long-Tail Bias in HOI Detection via Adaptive Diversity Cache](https://mlsanity.com/paper/2511.18811/) (2025)
- [Zero-shot HOI Detection with MLLM-based Detector-agnostic Interaction Recognition](https://mlsanity.com/paper/2602.15124/) (2026)
- [SHOE: Semantic HOI Open-Vocabulary Evaluation Metric](https://mlsanity.com/paper/2604.01586/) (2026)
- [Mining Instance-Centric Vision-Language Contexts for Human-Object Interaction Detection](https://mlsanity.com/paper/2604.02071/) (2026)
- [Unleashing Multimodal Large Language Models for Training-free HOI Detection in the Wild](https://mlsanity.com/paper/2607.13881/) (2026)
- [Exploring Interactive Semantic Alignment for Efficient HOI Detection with Vision-language Model](https://mlsanity.com/paper/2404.12678/) (2024)
- [Multimodal Graph Network Modeling for Human-Object Interaction Detection with PDE Graph Diffusion](https://mlsanity.com/paper/2509.12554/) (2025)

## Abstract (arXiv)

Human Object Interaction (HOI) detection aims to localize and infer the relationships between a human and an object. Arguably, training supervised models for this task from scratch presents challenges due to the performance drop over rare classes and the high computational cost and time required to handle long-tailed distributions of HOIs in complex HOI scenes in realistic settings. This observation motivates us to design an HOI detector that can be trained even with long-tailed labeled data and can leverage existing knowledge from pre-trained models. Inspired by the powerful generalization ability of the large Vision-Language Models (VLM) on classification and retrieval tasks, we propose an efficient Adaptive HOI Detector with Concept-guided Memory (ADA-CM). ADA-CM has two operating modes. The first mode makes it tunable without learning new parameters in a training-free paradigm. Its second mode incorporates an instance-aware adapter mechanism that can further efficiently boost performance if updating a lightweight set of parameters can be afforded. Our proposed method achieves competitive results with state-of-the-art on the HICO-DET and V-COCO datasets with much less training time. Code can be found at https://github.com/ltttpku/ADA-CM.