mlsanityData to 14 Oct 2026

Towards Unconstrained Human-Object Interaction

arXiv 2604.14069 ↗First posted 15 Apr 2026cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Human-Object Interaction (HOI) detection is a longstanding computer vision problem concerned with predicting the interaction between humans and objects. Current HOI models rely on a vocabulary of interactions at training and inference time, limiting their applicability to static environments. With the advent of Multimodal Large Language Models (MLLMs), it has become feasible to explore more flexible paradigms for interaction recognition. In this work, we revisit HOI detection through the lens of MLLMs and apply them to in-the-wild HOI detection. We define the Unconstrained HOI (U-HOI) task, a novel HOI domain that removes the requirement for a predefined list of interactions at both training and inference. We evaluate a range of MLLMs on this setting and introduce a pipeline that includes test-time inference and language-to-graph conversion to extract structured interactions from free-form text. Our findings highlight the limitations of current HOI detectors and the value of MLLMs for U-HOI. Code will be available at https://github.com/francescotonini/anyhoi

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 59 of its 74 references that are in the map. First posted on arXiv 15 Apr 2026.

Builds on

59 of 74 references.

References · 59

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  6. 2023Visual Instruction Tuningcited by 7,869
  7. 2020End-to-End Object Detection with Transformerscited by 7,414
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  10. 2024GPT-4o System Cardcited by 4,588
  11. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  12. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
38 more
  1. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  2. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  3. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  4. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  5. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  6. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  7. 2025s1: Simple test-time scalingcited by 1,341
  8. 2016MOT16: A Benchmark for Multi-Object Trackingcited by 532
  9. 2021TrackFormer: Multi-Object Tracking with Transformerscited by 518
  10. 2024What matters when building vision-language models?cited by 326
  11. 2017Learning to Detect Human-Object Interactionscited by 240
  12. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 218
  13. 2024CogVLM2: Visual Language Models for Image and Video Understandingcited by 217
  14. 2021HOTR: End-to-End Human-Object Interaction Detection with Transformerscited by 183
  15. 2019PPDM: Parallel Point Detection and Matching for Real-time Human-Object Interaction Detectioncited by 143
  16. 2022GEN-VLKT: Simplify Association and Enhance Interaction Understanding for HOI Detectioncited by 120
  17. 2020DRG: Dual Relation Graph for Human-Object Interaction Detectioncited by 117
  18. 2020VSGNet: Spatial Attention Network for Detecting Human Object Interactions Using Graph Convolutionscited by 115
  19. 2020Detailed 2D-3D Joint Representation for Human-Object Interactioncited by 93
  20. 2020Spatially Conditioned Graphs for Detecting Human-Object Interactionscited by 93
  21. 2021Efficient Two-Stage Detection of Human-Object Interactions with a Novel Unary-Pairwise Transformercited by 91
  22. 2023Verbs in Action: Improving verb understanding in video-language modelscited by 76
  23. 2020Detecting Human-Object Interactions with Action Co-occurrence Priorscited by 72
  24. 2023HOICLIP: Efficient Knowledge Transfer for HOI Detection with Vision-Language Modelscited by 61
  25. 2023Exploring Predicate Visual Context in Detecting Human-Object Interactionscited by 56
  26. 2021Opening up Open-World Trackingcited by 50
  27. 2023FACTUAL: A Benchmark for Faithful and Consistent Textual Scene Graph Parsingcited by 50
  28. 2022Exploring Structure-aware Transformer over Interaction Proposals for Human-Object Interaction Detectioncited by 47
  29. 2020ConsNet: Learning Consistency Graph for Zero-Shot Human-Object Interaction Detectioncited by 46
  30. 2023ViPLO: Vision Transformer based Pose-Conditioned Self-Loop Graph for Human-Object Interaction Detectioncited by 41
  31. 2023Efficient Adaptive Human-Object Interaction Detection with Concept-guided Memorycited by 38
  32. 2024Asynchronous Large Language Model Enhanced Planner for Autonomous Drivingcited by 38
  33. 2023UnionDet: Union-Level Detector Towards Real-Time Human-Object Interaction Detectioncited by 31
  34. 2022Revisiting Open World Object Detectioncited by 30
  35. 2022End-to-End Zero-Shot HOI Detection via Vision and Language Knowledge Distillationcited by 30
  36. 2023Visual Instruction Tuning towards General-Purpose Multimodal Model: A Surveycited by 29
  37. 2024Open-World Human-Object Interaction Detection via Multi-modal Promptscited by 28
  38. 2024EZ-HOI: VLM Adaptation via Guided Prompt Learning for Zero-Shot HOI Detectioncited by 28

50 most cited of 59 All 59 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Enhancing HOI Detection with Contextual Cues from Large Vision-Language Models
  2. 2025HOI-R1: Exploring the Potential of Multimodal Large Language Models for Human-Object Interaction Detection
  3. 2024VLM-HOI: Vision Language Models for Interpretable Human-Object Interaction Analysis
  4. 2026Harnessing Multimodal Large Language Models for Training-Free Human-Object Interaction Detection
  5. 2023Weakly-Supervised HOI Detection from Interaction Labels Only and Language/Vision-Language Priors
  6. 2026Streamlined Open-Vocabulary Human-Object Interaction Detection
  7. 2024CL-HOI: Cross-Level Human-Object Interaction Distillation from Vision Large Language Models
  8. 2026HOIBlender: Blending Lightweight Detection with Vision-Language Priors for Efficient Human-Object Interaction Detection