# Prototype Embedding Optimization for Human-Object Interaction Detection in Livestreaming

arXiv: https://arxiv.org/abs/2505.22011 · HTML: https://mlsanity.com/paper/2505.22011/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 22 references that are in the map.
- First posted on arXiv 28 May 2025.

## Builds on (most cited first, 13 of 13)

- [Simple Online and Realtime Tracking with a Deep Association Metric](https://mlsanity.com/paper/1703.07402/) (2017)
- [CrowdHuman: A Benchmark for Detecting Human in a Crowd](https://mlsanity.com/paper/1805.00123/) (2018)
- [Learning Human-Object Interactions by Graph Parsing Neural Networks](https://mlsanity.com/paper/1808.07962/) (2018)
- [QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Information](https://mlsanity.com/paper/2103.05399/) (2021)
- [Prototype Rectification for Few-Shot Learning](https://mlsanity.com/paper/1911.10713/) (2019)
- [Spatial-Temporal Transformer for Dynamic Scene Graph Generation](https://mlsanity.com/paper/2107.12309/) (2021)
- [Detecting Attended Visual Targets in Video](https://mlsanity.com/paper/2003.02501/) (2020)
- [ST-HOI: A Spatial-Temporal Baseline for Human-Object Interaction Detection in Videos](https://mlsanity.com/paper/2105.11731/) (2021)
- [LIGHTEN: Learning Interactions with Graph and Hierarchical TEmporal Networks for HOI in videos](https://mlsanity.com/paper/2012.09402/) (2020)
- [Video-based Human-Object Interaction Detection from Tubelet Tokens](https://mlsanity.com/paper/2206.01908/) (2022)
- [Human-Object Interaction Prediction in Videos through Gaze Following](https://mlsanity.com/paper/2306.03597/) (2023)
- [Highlighting Object Category Immunity for the Generalization of Human-Object Interaction Detection](https://mlsanity.com/paper/2202.09492/) (2022)
- [Chairs Can be Stood on: Overcoming Object Bias in Human-Object Interaction Detection](https://mlsanity.com/paper/2207.02400/) (2022)

## Abstract (arXiv)

Livestreaming often involves interactions between streamers and objects, which is critical for understanding and regulating web content. While human-object interaction (HOI) detection has made some progress in general-purpose video downstream tasks, when applied to recognize the interaction behaviors between a streamer and different objects in livestreaming, it tends to focuses too much on the objects and neglects their interactions with the streamer, which leads to object bias. To solve this issue, we propose a prototype embedding optimization for human-object interaction detection (PeO-HOI). First, the livestreaming is preprocessed using object detection and tracking techniques to extract features of the human-object (HO) pairs. Then, prototype embedding optimization is adopted to mitigate the effect of object bias on HOI. Finally, after modelling the spatio-temporal context between HO pairs, the HOI detection results are obtained by the prediction head. The experimental results show that the detection accuracy of the proposed PeO-HOI method has detection accuracies of 37.19%@full, 51.42%@non-rare, 26.20%@rare on the publicly available dataset VidHOI, 45.13%@full, 62.78%@non-rare and 30.37%@rare on the self-built dataset BJUT-HOI, which effectively improves the HOI detection performance in livestreaming.