# Single-Query Person-Centric Bimanual Hand-Object Interaction Detection

arXiv: https://arxiv.org/abs/2609.12155 · HTML: https://mlsanity.com/paper/2609.12155/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 31 references that are in the map.
- First posted on arXiv 10 Sep 2026.

## Builds on (most cited first, 25 of 25)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Depth Anything V2](https://mlsanity.com/paper/2406.09414/) (2024)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection](https://mlsanity.com/paper/2203.03605/) (2022)
- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [DETRs Beat YOLOs on Real-time Object Detection](https://mlsanity.com/paper/2304.08069/) (2023)
- [Conflict-Averse Gradient Descent for Multi-task Learning](https://mlsanity.com/paper/2110.14048/) (2021)
- [Understanding Human Hands in Contact at Internet Scale](https://mlsanity.com/paper/2006.06669/) (2020)
- [DETRs with Collaborative Hybrid Assignments Training](https://mlsanity.com/paper/2211.12860/) (2022)
- [Learning to Detect Human-Object Interactions](https://mlsanity.com/paper/1702.05448/) (2017)
- [Visual Semantic Role Labeling](https://mlsanity.com/paper/1505.04474/) (2015)
- [HOTR: End-to-End Human-Object Interaction Detection with Transformers](https://mlsanity.com/paper/2104.13682/) (2021)
- [EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relations](https://mlsanity.com/paper/2209.13064/) (2022)
- [RelTR: Relation Transformer for Scene Graph Generation](https://mlsanity.com/paper/2201.11460/) (2022)
- [YOLO-Pose: Enhancing YOLO for Multi Person Pose Estimation Using Object Keypoint Similarity Loss](https://mlsanity.com/paper/2204.06806/) (2022)
- [Exploring Predicate Visual Context in Detecting Human-Object Interactions](https://mlsanity.com/paper/2308.06202/) (2023)
- [Explicit Box Detection Unifies End-to-End Multi-Person Pose Estimation](https://mlsanity.com/paper/2302.01593/) (2023)
- [EGTR: Extracting Graph from Transformer for Scene Graph Generation](https://mlsanity.com/paper/2404.02072/) (2024)
- [Human Pose as Compositional Tokens](https://mlsanity.com/paper/2303.11638/) (2023)
- [Detecting Hands and Recognizing Physical Contact in the Wild](https://mlsanity.com/paper/2010.09676/) (2020)
- [Open-World Human-Object Interaction Detection via Multi-modal Prompts](https://mlsanity.com/paper/2406.07221/) (2024)
- [Group Pose: A Simple Baseline for End-to-End Multi-person Pose Estimation](https://mlsanity.com/paper/2308.07313/) (2023)
- [Open-Vocabulary HOI Detection with Interaction-aware Prompt and Concept Calibration](https://mlsanity.com/paper/2508.03207/) (2025)

## Abstract (arXiv)

Understanding person-level bi-manual interactions requires not only detecting hands, but also identifying which two hands belong to the same person and what each hand interacts with. Existing hand--object interaction methods are mostly hand-centric: they treat each hand as an independent instance, which can lead to ambiguous ownership in multi-person scenes. We propose a person-centric formulation in which a single query predicts a structured output for one person, including the human box, body pose, hand boxes and states, and interaction targets. We introduce part-aware deformable attention to allocate attention across human, hand, and pose-specific reference regions, enabling one query to capture the full person structure. We further unify detection and interaction reasoning with a hand-to-query relationship matrix, where each hand selects its interaction target from the detected query set plus a learnable off token, directly recovering the target's box and class without separate object regression. We build a COCO-based dataset with person-centric bi-manual interaction annotations and define structured metrics for evaluating hand states and complete hand--object tuples. Experiments with a transformer-based detector show that our formulation improves person-level bi-manual interaction parsing and provides an effective unified framework for joint detection, pose estimation, and hand reasoning.