# Consistency Learning via Decoding Path Augmentation for Transformers in Human Object Interaction Detection

arXiv: https://arxiv.org/abs/2204.04836 · HTML: https://mlsanity.com/paper/2204.04836/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 51 of its 63 references that are in the map.
- First posted on arXiv 11 Apr 2022.

## Builds on (most cited first, 30 of 51)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [MixMatch: A Holistic Approach to Semi-Supervised Learning](https://mlsanity.com/paper/1905.02249/) (2019)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [Unsupervised Data Augmentation for Consistency Training](https://mlsanity.com/paper/1904.12848/) (2019)
- [Temporal Ensembling for Semi-Supervised Learning](https://mlsanity.com/paper/1610.02242/) (2016)
- [Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning](https://mlsanity.com/paper/1704.03976/) (2017)
- [Stacked Hourglass Networks for Human Pose Estimation](https://mlsanity.com/paper/1603.06937/) (2016)
- [Deep Mutual Learning](https://mlsanity.com/paper/1706.00384/) (2017)
- [Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results](https://mlsanity.com/paper/1703.01780/) (2017)
- [Conditional DETR for Fast Training Convergence](https://mlsanity.com/paper/2108.06152/) (2021)
- [Cascaded Pyramid Network for Multi-Person Pose Estimation](https://mlsanity.com/paper/1711.07319/) (2017)
- [Learning to Detect Human-Object Interactions](https://mlsanity.com/paper/1702.05448/) (2017)
- [CrowdPose: Efficient Crowded Scenes Pose Estimation and A New Benchmark](https://mlsanity.com/paper/1812.00324/) (2018)
- [Detecting and Recognizing Human-Object Interactions](https://mlsanity.com/paper/1704.07333/) (2017)
- [Visual Semantic Role Labeling](https://mlsanity.com/paper/1505.04474/) (2015)
- [HOTR: End-to-End Human-Object Interaction Detection with Transformers](https://mlsanity.com/paper/2104.13682/) (2021)
- [Learning Human-Object Interactions by Graph Parsing Neural Networks](https://mlsanity.com/paper/1808.07962/) (2018)
- [Fast Convergence of DETR with Spatially Modulated Co-Attention](https://mlsanity.com/paper/2101.07448/) (2021)
- [Temporal Cycle-Consistency Learning](https://mlsanity.com/paper/1904.07846/) (2019)
- [QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Information](https://mlsanity.com/paper/2103.05399/) (2021)
- [PPDM: Parallel Point Detection and Matching for Real-time Human-Object Interaction Detection](https://mlsanity.com/paper/1912.12898/) (2019)

## Built on by (most cited first, 16 of 16)

- [RLIPv2: Fast Scaling of Relational Language-Image Pre-training](https://mlsanity.com/paper/2308.09351/) (2023)
- [TokenMixup: Efficient Attention-guided Token-level Data Augmentation for Transformers](https://mlsanity.com/paper/2210.07562/) (2022)
- [Category Query Learning for Human-Object Interaction Classification](https://mlsanity.com/paper/2303.14005/) (2023)
- [Human-Object Interaction Detection Collaborated with Large Relation-driven Diffusion Models](https://mlsanity.com/paper/2410.20155/) (2024)
- [Unified Visual Relationship Detection with Vision and Language Models](https://mlsanity.com/paper/2303.08998/) (2023)
- [Human-Object Interaction Prediction in Videos through Gaze Following](https://mlsanity.com/paper/2306.03597/) (2023)
- [Groupwise Query Specialization and Quality-Aware Multi-Assignment for Transformer-based Visual Relationship Detection](https://mlsanity.com/paper/2403.17709/) (2024)
- [Improving Human-Object Interaction Detection via Virtual Image Learning](https://mlsanity.com/paper/2308.02606/) (2023)
- [DDS: Decoupled Dynamic Scene-Graph Generation Network](https://mlsanity.com/paper/2301.07666/) (2023)
- [Visual Diversity and Region-aware Prompt Learning for Zero-shot HOI Detection](https://mlsanity.com/paper/2510.25094/) (2025)
- [TriangleNet: Edge Prior Augmented Network for Semantic Segmentation through Cross-Task Consistency](https://mlsanity.com/paper/2210.05152/) (2022)
- [SGTR+: End-to-end Scene Graph Generation with Transformer](https://mlsanity.com/paper/2401.12835/) (2024)
- [Mitigating Long-Tail Bias in HOI Detection via Adaptive Diversity Cache](https://mlsanity.com/paper/2511.18811/) (2025)
- [ScriptHOI: Learning Scripted State Transitions for Open-Vocabulary Human-Object Interaction Detection](https://mlsanity.com/paper/2605.05057/) (2026)
- [STMixer: A One-Stage Sparse Action Detector](https://mlsanity.com/paper/2404.09842/) (2024)
- [RegFormer: Transferable Relational Grounding for Efficient Weakly-Supervised Human-Object Interaction Detection](https://mlsanity.com/paper/2604.00507/) (2026)

## Abstract (arXiv)

Human-Object Interaction detection is a holistic visual recognition task that entails object detection as well as interaction classification. Previous works of HOI detection has been addressed by the various compositions of subset predictions, e.g., Image -> HO -> I, Image -> HI -> O. Recently, transformer based architecture for HOI has emerged, which directly predicts the HOI triplets in an end-to-end fashion (Image -> HOI). Motivated by various inference paths for HOI detection, we propose cross-path consistency learning (CPC), which is a novel end-to-end learning strategy to improve HOI detection for transformers by leveraging augmented decoding paths. CPC learning enforces all the possible predictions from permuted inference sequences to be consistent. This simple scheme makes the model learn consistent representations, thereby improving generalization without increasing model capacity. Our experiments demonstrate the effectiveness of our method, and we achieved significant improvement on V-COCO and HICO-DET compared to the baseline models. Our code is available at https://github.com/mlvlab/CPChoi.