# EgoInteract: Synthetic Egocentric Videos Generation for Interaction Understanding and Anticipation

arXiv: https://arxiv.org/abs/2605.18214 · HTML: https://mlsanity.com/paper/2605.18214/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 49 references that are in the map.
- First posted on arXiv 18 May 2026.

## Builds on (most cited first, 30 of 38)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [AI2-THOR: An Interactive 3D Environment for Visual AI](https://mlsanity.com/paper/1712.05474/) (2017)
- [Scaling Egocentric Vision: The EPIC-KITCHENS Dataset](https://mlsanity.com/paper/1804.02748/) (2018)
- [Objaverse-XL: A Universe of 10M+ 3D Objects](https://mlsanity.com/paper/2307.05663/) (2023)
- [Rescaling Egocentric Vision](https://mlsanity.com/paper/2006.13256/) (2020)
- [Habitat 2.0: Training Home Assistants to Rearrange their Habitat](https://mlsanity.com/paper/2106.14405/) (2021)
- [Temporal Convolutional Networks for Action Segmentation and Detection](https://mlsanity.com/paper/1611.05267/) (2016)
- [Gibson Env: Real-World Perception for Embodied Agents](https://mlsanity.com/paper/1808.10654/) (2018)
- [PointRend: Image Segmentation as Rendering](https://mlsanity.com/paper/1912.08193/) (2019)
- [MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation](https://mlsanity.com/paper/1903.01945/) (2019)
- [Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities](https://mlsanity.com/paper/2203.14712/) (2022)
- [Understanding Human Hands in Contact at Internet Scale](https://mlsanity.com/paper/2006.06669/) (2020)
- [ASFormer: Transformer for Action Segmentation](https://mlsanity.com/paper/2110.08568/) (2021)
- [EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relations](https://mlsanity.com/paper/2209.13064/) (2022)
- [Grounded Human-Object Interaction Hotspots from Video](https://mlsanity.com/paper/1812.04558/) (2018)
- [Cross-Domain Adaptive Teacher for Object Detection](https://mlsanity.com/paper/2111.13216/) (2021)
- [HD-EPIC: A Highly-Detailed Egocentric Video Dataset](https://mlsanity.com/paper/2502.04144/) (2025)
- [SceneScript: Reconstructing Scenes With An Autoregressive Structured Language Model](https://mlsanity.com/paper/2403.13064/) (2024)
- [MAISI: Medical AI for Synthetic Imaging](https://mlsanity.com/paper/2409.11169/) (2024)
- [The MECCANO Dataset: Understanding Human-Object Interactions from Egocentric Videos in an Industrial-like Domain](https://mlsanity.com/paper/2010.05654/) (2020)
- [Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications](https://mlsanity.com/paper/2208.03826/) (2022)
- [MOTSynth: How Can Synthetic Data Help Pedestrian Detection and Tracking?](https://mlsanity.com/paper/2108.09518/) (2021)
- [Next-Active-Object prediction from Egocentric Videos](https://mlsanity.com/paper/1904.05250/) (2019)
- [MECCANO: A Multimodal Egocentric Dataset for Humans Behavior Understanding in the Industrial-like Domain](https://mlsanity.com/paper/2209.08691/) (2022)
- [Coarse to Fine Multi-Resolution Temporal Convolutional Network](https://mlsanity.com/paper/2105.10859/) (2021)

## Built on by (most cited first, 1 of 1)

- [G3Ego: Gaze-Guided Graphs for Egocentric Action Understanding](https://mlsanity.com/paper/2608.20157/) (2026)

## Abstract (arXiv)

Collecting large-scale egocentric video datasets with dense spatial and temporal annotations is costly, slow, and often constrained by environmental biases, privacy constraints, and limited coverage of interaction patterns. While synthetic data has shown strong potential in several vision domains, its use for egocentric perception remains relatively underexplored, especially for tasks requiring temporally coherent human-object interactions. In this work, we introduce EgoInteract, a controllable simulator for egocentric video generation designed to model fine-grained egocentric interactions and their temporal dynamics. The simulator enables precise control over camera, human body and hand motion, object manipulation, and scene composition across diverse environments. Building on this framework, we generate a synthetic egocentric video dataset with dense spatial and temporal annotations for temporal action segmentation, next-active object detection, interaction anticipation, and hand-object interaction detection. We evaluate models trained with simulated data on multiple real-world egocentric benchmarks spanning diverse environments, object categories, and interaction patterns. Results show consistent improvements over strong baselines across tasks and datasets, demonstrating the effectiveness and transferability of our simulation-based approach.