# Identity-Aware Human-Object Interaction Motion Captioning

arXiv: https://arxiv.org/abs/2608.20690 · HTML: https://mlsanity.com/paper/2608.20690/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 31 references that are in the map.
- First posted on arXiv 21 Aug 2026.

## Builds on (most cited first, 17 of 17)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](https://mlsanity.com/paper/2606.19348/) (2026)
- [Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning](https://mlsanity.com/paper/2302.14115/) (2023)
- [BEHAVE: Dataset and Method for Tracking Human Object Interactions](https://mlsanity.com/paper/2204.06950/) (2022)
- [SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioning](https://mlsanity.com/paper/2111.13196/) (2021)
- [CLIP4Caption: CLIP for Video Caption](https://mlsanity.com/paper/2110.06615/) (2021)
- [Combining detection and tracking for human pose estimation in videos](https://mlsanity.com/paper/2003.13743/) (2020)
- [Attend and Interact: Higher-Order Object Interactions for Video Understanding](https://mlsanity.com/paper/1711.06330/) (2017)
- [InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generation](https://mlsanity.com/paper/2509.09555/) (2025)
- [Accurate and Fast Compressed Video Captioning](https://mlsanity.com/paper/2309.12867/) (2023)
- [OmniVid: A Generative Framework for Universal Video Understanding](https://mlsanity.com/paper/2403.17935/) (2024)
- [Identity-Aware Multi-Sentence Video Description](https://mlsanity.com/paper/2008.09791/) (2020)
- [CLIP Meets Video Captioning: Concept-Aware Representation Learning Does Matter](https://mlsanity.com/paper/2111.15162/) (2021)
- [MICap: A Unified Model for Identity-aware Movie Descriptions](https://mlsanity.com/paper/2405.11483/) (2024)
- [SAVCHOI: Detecting Suspicious Activities using Dense Video Captioning with Human Object Interactions](https://mlsanity.com/paper/2207.11838/) (2022)
- [Towards Fine-Grained Human Motion Video Captioning](https://mlsanity.com/paper/2510.24767/) (2025)

## Abstract (arXiv)

Existing human-object interaction (HOI) motion captioning methods typically describe what happens while referring to the subject using generic terms such as "a person" or "someone", without grounding the caption in subject identity. To address this limitation, we introduce Identity-Aware Human-Object Interaction Motion Captioning task. This task requires each generated caption to specify both the subject identity and the corresponding HOI motion. For example, the model generates "Sub_ID lifts the chair" rather than "A person lifts the chair". For this task, we design identity-aware HOI motion captions based on the BEHAVE and InterCap datasets. We further propose ID-HOINet, which learns from multi-view videos while supporting single-view identity-aware HOI motion caption generation. ID-HOINet contains two core components: Multi-View Identity-Motion Learning Module (MVIML) and Two-Stage Caption Rewriting Strategy (TSCR). MVIML learns from multi-view videos by modeling dependencies across temporal stages and camera viewpoints, capturing identity and interaction motion features. At inference, the TSCR first retrieves the subject identity and generates identity-agnostic HOI motion captions. TSCR then rewrites these captions with the predicted identity to produce the final identity-aware HOI motion captions. Experiments demonstrate that ID-HOINet achieves state-of-the-art performance. Code will be released upon acceptance.