# HOI4ABOT: Human-Object Interaction Anticipation for Human Intention Reading Collaborative roBOTs

arXiv: https://arxiv.org/abs/2309.16524 · HTML: https://mlsanity.com/paper/2309.16524/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 57 references that are in the map.
- First posted on arXiv 28 Sep 2023.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [Class-Balanced Loss Based on Effective Number of Samples](https://mlsanity.com/paper/1901.05555/) (2019)
- [ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation](https://mlsanity.com/paper/2204.12484/) (2022)
- [Image Captioning with Semantic Attention](https://mlsanity.com/paper/1603.03925/) (2016)
- [Action Genome: Actions as Composition of Spatio-temporal Scene Graphs](https://mlsanity.com/paper/1912.06992/) (2019)
- [Visual Semantic Role Labeling](https://mlsanity.com/paper/1505.04474/) (2015)
- [QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Information](https://mlsanity.com/paper/2103.05399/) (2021)
- [GEN-VLKT: Simplify Association and Enhance Interaction Understanding for HOI Detection](https://mlsanity.com/paper/2203.13954/) (2022)
- [VSGNet: Spatial Attention Network for Detecting Human Object Interactions Using Graph Convolutions](https://mlsanity.com/paper/2003.05541/) (2020)
- [Behavior Trees in Robotics and AI: An Introduction](https://mlsanity.com/paper/1709.00084/) (2017)
- [Spatial-Temporal Transformer for Dynamic Scene Graph Generation](https://mlsanity.com/paper/2107.12309/) (2021)
- [TASKOGRAPHY: Evaluating robot task planning over large 3D scene graphs](https://mlsanity.com/paper/2207.05006/) (2022)
- [ViPLO: Vision Transformer based Pose-Conditioned Self-Loop Graph for Human-Object Interaction Detection](https://mlsanity.com/paper/2304.08114/) (2023)
- [Intention-Conditioned Long-Term Human Egocentric Action Forecasting](https://mlsanity.com/paper/2207.12080/) (2022)
- [Mining Cross-Person Cues for Body-Part Interactiveness Learning in HOI Detection](https://mlsanity.com/paper/2207.14192/) (2022)
- [Object Handovers: a Review for Robotics](https://mlsanity.com/paper/2007.12952/) (2020)
- [Reasoning with Scene Graphs for Robot Planning under Partial Observability](https://mlsanity.com/paper/2202.10432/) (2022)
- [Robotic Vision for Human-Robot Interaction and Collaboration: A Survey and Systematic Review](https://mlsanity.com/paper/2307.15363/) (2023)
- [CoSTAR: Instructing Collaborative Robots with Behavior Trees and Vision](https://mlsanity.com/paper/1611.06145/) (2016)
- [ST-HOI: A Spatial-Temporal Baseline for Human-Object Interaction Detection in Videos](https://mlsanity.com/paper/2105.11731/) (2021)
- [Human-robot collaboration and machine learning: a systematic review of recent research](https://mlsanity.com/paper/2110.07448/) (2021)
- [Proactive Robot Assistance via Spatio-Temporal Object Modeling](https://mlsanity.com/paper/2211.15501/) (2022)
- [Video-based Human-Object Interaction Detection from Tubelet Tokens](https://mlsanity.com/paper/2206.01908/) (2022)
- [Human-Object Interaction Prediction in Videos through Gaze Following](https://mlsanity.com/paper/2306.03597/) (2023)

## Built on by (most cited first, 12 of 12)

- [HUMOTO: A 4D Dataset of Mocap Human Object Interactions](https://mlsanity.com/paper/2504.10414/) (2025)
- [Perceiving and Acting in First-Person: A Dataset and Benchmark for Egocentric Human-Object-Human Interactions](https://mlsanity.com/paper/2508.04681/) (2025)
- [An Image-like Diffusion Method for Human-Object Interaction Detection](https://mlsanity.com/paper/2503.18134/) (2025)
- [RHINO: Learning Real-Time Humanoid-Human-Object Interaction from Human Demonstrations](https://mlsanity.com/paper/2502.13134/) (2025)
- [SGC-Net: Stratified Granular Comparison Network for Open-Vocabulary HOI Detection](https://mlsanity.com/paper/2503.00414/) (2025)
- [Robot Behavior Personalization from Sparse User Feedback](https://mlsanity.com/paper/2410.19219/) (2024)
- [Foundation Model-Driven Framework for Human-Object Interaction Prediction with Segmentation Mask Integration](https://mlsanity.com/paper/2504.19847/) (2025)
- [Incremental Human-Object Interaction Detection with Invariant Relation Representation Learning](https://mlsanity.com/paper/2510.27020/) (2025)
- [Generative Human-Object Interaction Detection via Differentiable Cognitive Steering of Multi-modal LLMs](https://mlsanity.com/paper/2512.17640/) (2025)
- [Towards Unconstrained Human-Object Interaction](https://mlsanity.com/paper/2604.14069/) (2026)
- [SAGE: Synchronized Action-Gaze Recognition and Anticipation for Human Behavior Understanding](https://mlsanity.com/paper/2607.04017/) (2026)
- [RGB-D Video Generation for Improving Human-to-Robot Object Handover Prediction](https://mlsanity.com/paper/2608.13028/) (2026)

## Abstract (arXiv)

Robots are becoming increasingly integrated into our lives, assisting us in various tasks. To ensure effective collaboration between humans and robots, it is essential that they understand our intentions and anticipate our actions. In this paper, we propose a Human-Object Interaction (HOI) anticipation framework for collaborative robots. We propose an efficient and robust transformer-based model to detect and anticipate HOIs from videos. This enhanced anticipation empowers robots to proactively assist humans, resulting in more efficient and intuitive collaborations. Our model outperforms state-of-the-art results in HOI detection and anticipation in VidHOI dataset with an increase of 1.76% and 1.04% in mAP respectively while being 15.4 times faster. We showcase the effectiveness of our approach through experimental results in a real robot, demonstrating that the robot's ability to anticipate HOIs is key for better Human-Robot Interaction. More information can be found on our project webpage: https://evm7.github.io/HOI4ABOT_page/