# On the Robustness of Human-Object Interaction Detection against Distribution Shift

arXiv: https://arxiv.org/abs/2506.18021 · HTML: https://mlsanity.com/paper/2506.18021/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 61 of its 77 references that are in the map.
- First posted on arXiv 22 Jun 2025.

## Builds on (most cited first, 30 of 61)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Null-text Inversion for Editing Real Images using Guided Diffusion Models](https://mlsanity.com/paper/2211.09794/) (2022)
- [How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites](https://mlsanity.com/paper/2404.16821/) (2024)
- [EVA-CLIP: Improved Training Techniques for CLIP at Scale](https://mlsanity.com/paper/2303.15389/) (2023)
- [EVA: Exploring the Limits of Masked Visual Representation Learning at Scale](https://mlsanity.com/paper/2211.07636/) (2022)
- [CenterNet: Keypoint Triplets for Object Detection](https://mlsanity.com/paper/1904.08189/) (2019)
- [DeiT III: Revenge of the ViT](https://mlsanity.com/paper/2204.07118/) (2022)
- [EVA-02: A Visual Representation for Neon Genesis](https://mlsanity.com/paper/2303.11331/) (2023)
- [Noise or Signal: The Role of Image Backgrounds in Object Recognition](https://mlsanity.com/paper/2006.09994/) (2020)
- [Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming](https://mlsanity.com/paper/1907.07484/) (2019)
- [Learning to Detect Human-Object Interactions](https://mlsanity.com/paper/1702.05448/) (2017)
- [Concrete Dropout](https://mlsanity.com/paper/1705.07832/) (2017)
- [Visual Semantic Role Labeling](https://mlsanity.com/paper/1505.04474/) (2015)
- [QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Information](https://mlsanity.com/paper/2103.05399/) (2021)
- [PPDM: Parallel Point Detection and Matching for Real-time Human-Object Interaction Detection](https://mlsanity.com/paper/1912.12898/) (2019)
- [GEN-VLKT: Simplify Association and Enhance Interaction Understanding for HOI Detection](https://mlsanity.com/paper/2203.13954/) (2022)

## Abstract (arXiv)

Human-Object Interaction (HOI) detection has seen substantial advances in recent years. However, existing works focus on the standard setting with ideal images and natural distribution, far from practical scenarios with inevitable distribution shifts. This hampers the practical applicability of HOI detection. In this work, we investigate this issue by benchmarking, analyzing, and enhancing the robustness of HOI detection models under various distribution shifts. We start by proposing a novel automated approach to create the first robustness evaluation benchmark for HOI detection. Subsequently, we evaluate more than 40 existing HOI detection models on this benchmark, showing their insufficiency, analyzing the features of different frameworks, and discussing how the robustness in HOI is different from other tasks. With the insights from such analyses, we propose to improve the robustness of HOI detection methods through: (1) a cross-domain data augmentation integrated with mixup, and (2) a feature fusion strategy with frozen vision foundation models. Both are simple, plug-and-play, and applicable to various methods. Our experimental results demonstrate that the proposed approach significantly increases the robustness of various methods, with benefits on standard benchmarks, too. The dataset and code will be released.