# Are Synthetic Data Useful for Egocentric Hand-Object Interaction Detection?

arXiv: https://arxiv.org/abs/2312.02672 · HTML: https://mlsanity.com/paper/2312.02672/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 74 references that are in the map.
- First posted on arXiv 5 Dec 2023.

## Builds on (most cited first, 30 of 49)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop](https://mlsanity.com/paper/1506.03365/) (2015)
- [Adversarial Discriminative Domain Adaptation](https://mlsanity.com/paper/1702.05464/) (2017)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [A Comprehensive Survey on Transfer Learning](https://mlsanity.com/paper/1911.02685/) (2019)
- [AI2-THOR: An Interactive 3D Environment for Visual AI](https://mlsanity.com/paper/1712.05474/) (2017)
- [Scaling Egocentric Vision: The EPIC-KITCHENS Dataset](https://mlsanity.com/paper/1804.02748/) (2018)
- [Maximum Classifier Discrepancy for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1712.02560/) (2017)
- [Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Items](https://mlsanity.com/paper/2204.11918/) (2022)
- [Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI](https://mlsanity.com/paper/2109.08238/) (2021)
- [Rescaling Egocentric Vision](https://mlsanity.com/paper/2006.13256/) (2020)
- [Habitat 2.0: Training Home Assistants to Rearrange their Habitat](https://mlsanity.com/paper/2106.14405/) (2021)
- [Gibson Env: Real-World Perception for Embodied Agents](https://mlsanity.com/paper/1808.10654/) (2018)
- [PointRend: Image Segmentation as Rendering](https://mlsanity.com/paper/1912.08193/) (2019)
- [Learning joint reconstruction of hands and manipulated objects](https://mlsanity.com/paper/1904.05767/) (2019)
- [Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks](https://mlsanity.com/paper/1612.05424/) (2016)
- [Unbiased Teacher for Semi-Supervised Object Detection](https://mlsanity.com/paper/2102.09480/) (2021)
- [Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities](https://mlsanity.com/paper/2203.14712/) (2022)
- [Understanding Human Hands in Contact at Internet Scale](https://mlsanity.com/paper/2006.06669/) (2020)
- [iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Tasks](https://mlsanity.com/paper/2108.03272/) (2021)
- [Unbiased Mean Teacher for Cross-domain Object Detection](https://mlsanity.com/paper/2003.00707/) (2020)
- [EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relations](https://mlsanity.com/paper/2209.13064/) (2022)
- [Exploring Object Relation in Mean Teacher for Cross-Domain Detection](https://mlsanity.com/paper/1904.11245/) (2019)
- [DexGraspNet: A Large-Scale Robotic Dexterous Grasp Dataset for General Objects Based on Simulation](https://mlsanity.com/paper/2210.02697/) (2022)
- [Domain Adaptation for Visual Applications: A Comprehensive Survey](https://mlsanity.com/paper/1702.05374/) (2017)

## Built on by (most cited first, 10 of 10)

- [DiffH2O: Diffusion-Based Synthesis of Hand-Object Interactions from Textual Descriptions](https://mlsanity.com/paper/2403.17827/) (2024)
- [CaRe-Ego: Contact-aware Relationship Modeling for Egocentric Interactive Hand-object Segmentation](https://mlsanity.com/paper/2407.05576/) (2024)
- [Interaction-aware Representation Modeling with Co-occurrence Consistency for Egocentric Hand-Object Parsing](https://mlsanity.com/paper/2602.20597/) (2026)
- [FEEL (Force-Enhanced Egocentric Learning): A Dataset for Physical Action Understanding](https://mlsanity.com/paper/2603.15847/) (2026)
- [ENIGMA-360: An Ego-Exo Dataset for Human Behavior Understanding in Industrial Scenarios](https://mlsanity.com/paper/2603.09741/) (2026)
- [EgoInteract: Synthetic Egocentric Videos Generation for Interaction Understanding and Anticipation](https://mlsanity.com/paper/2605.18214/) (2026)
- [VR-based generation of photorealistic synthetic data for training hand-object tracking models](https://mlsanity.com/paper/2401.17874/) (2024)
- [Learning Egocentric In-Hand Object Segmentation through Weak Supervision from Human Narrations](https://mlsanity.com/paper/2509.26004/) (2025)
- [Leveraging Synthetic Data for Enhancing Egocentric Hand-Object Interaction Detection](https://mlsanity.com/paper/2603.29733/) (2026)
- [Removing Infrastructure Barriers in Human-Robot Collaboration Through Wireless Reconfigurable Cells](https://mlsanity.com/paper/2608.09658/) (2026)

## Abstract (arXiv)

In this study, we investigate the effectiveness of synthetic data in enhancing egocentric hand-object interaction detection. Via extensive experiments and comparative analyses on three egocentric datasets, VISOR, EgoHOS, and ENIGMA-51, our findings reveal how to exploit synthetic data for the HOI detection task when real labeled data are scarce or unavailable. Specifically, by leveraging only 10% of real labeled data, we achieve improvements in Overall AP compared to baselines trained exclusively on real data of: +5.67% on EPIC-KITCHENS VISOR, +8.24% on EgoHOS, and +11.69% on ENIGMA-51. Our analysis is supported by a novel data generation pipeline and the newly introduced HOI-Synth benchmark which augments existing datasets with synthetic images of hand-object interactions automatically labeled with hand-object contact states, bounding boxes, and pixel-wise segmentation masks. Data, code, and data generation tools to support future research are released at: https://fpv-iplab.github.io/HOI-Synth/.