mlsanityData to 14 Oct 2026

HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real World

arXiv 2309.17024 ↗First posted 29 Sep 2023cs.CVcited by 166

Not on the map. Why?

Cited by 22 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Building an interactive AI assistant that can perceive, reason, and collaborate with humans in the real world has been a long-standing pursuit in the AI community. This work is part of a broader research effort to develop intelligent agents that can interactively guide humans through performing tasks in the physical world. As a first step in this direction, we introduce HoloAssist, a large-scale egocentric human interaction dataset, where two people collaboratively complete physical manipulation tasks. The task performer executes the task while wearing a mixed-reality headset that captures seven synchronized data streams. The task instructor watches the performer's egocentric video in real time and guides them verbally. By augmenting the data with action and conversational annotations and observing the rich behaviors of various participants, we present key insights into how human assistants correct mistakes, intervene in the task completion procedure, and ground their instructions to the environment. HoloAssist spans 166 hours of data captured by 350 unique instructor-performer pairs. Furthermore, we construct and present benchmarks on mistake detection, intervention type prediction, and hand forecasting, along with detailed analysis. We expect HoloAssist will provide an important resource for building AI assistants that can fluidly collaborate with humans in the real world. Data can be downloaded at https://holoassist.github.io/.

Key facts. Cited by 166 papers in the map (data through 14 Oct 2026). Builds on 31 of its 44 references that are in the map. First posted on arXiv 29 Sep 2023.

Builds on

31 of 44 references.

References · 31

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  5. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  6. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  7. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  8. 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
  9. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 869
  10. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 783
  11. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 551
  12. 2020Rescaling Egocentric Visioncited by 540
19 more
  1. 2021Habitat 2.0: Training Home Assistants to Rearrange their Habitatcited by 511
  2. 2018VirtualHome: Simulating Household Activities via Programscited by 415
  3. 2022HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interactioncited by 327
  4. 2022Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activitiescited by 287
  5. 2021H2O: Two Hands Manipulating Objects for First Person Interaction Recognitioncited by 254
  6. 2019Vision-and-Dialog Navigationcited by 250
  7. 2021TEACh: Task-driven Embodied Agents that Chatcited by 197
  8. 2021BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environmentscited by 165
  9. 2021EgoBody: Human Body Shape and Motion of Interacting People from Head-Mounted Devicescited by 139
  10. 2015Unsupervised Learning from Narrated Instruction Videoscited by 116
  11. 2022Joint Hand Motion and Interaction Hotspots Prediction from Egocentric Videoscited by 111
  12. 2022GIMO: Gaze-Informed Human Motion Prediction in Contextcited by 81
  13. 2020The MECCANO Dataset: Understanding Human-Object Interactions from Egocentric Videos in an Industrial-like Domaincited by 79
  14. 2022TL;DW? Summarizing Instructional Videos with Task Relevance & Cross-Modal Saliencycited by 35
  15. 2022AssistQ: Affordance-centric Question-driven Task Completion for Egocentric Assistantcited by 32
  16. 2022Towards Open Set Video Anomaly Detectioncited by 26
  17. 2022Interactive Grounded Language Understanding in a Collaborative Environment: IGLU 2021cited by 23
  18. 2022IGLU Gridworld: Simple and Fast Environment for Embodied Dialog Agentscited by 9
  19. 2022Human in the loop approaches in multi-modal conversational task guidance system developmentcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 26; 2025: 62; 2026: 76.

166 papers.

Sort:

By area:

  1. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,225
  2. 2025Cosmos-Reason1: From Physical Common Sense To Embodied Reasoningcited by 164
  3. 2025EgoVLA: Learning Vision-Language-Action Models from Egocentric Human Videoscited by 128
  4. 2025HD-EPIC: A Highly-Detailed Egocentric Video Datasetcited by 104
  5. 2025Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videoscited by 102
  6. 2024EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real Worldcited by 95
  7. 2025EgoLife: Towards Egocentric Life Assistantcited by 91
  8. 2025villa-X: Enhancing Latent Action Modeling in Vision-Language-Action Modelscited by 63
  9. 2025OmniWorld: A Multi-Domain and Multi-Modal Dataset for 4D World Modelingcited by 62
  10. 2023CaptainCook4D: A Dataset for Understanding Errors in Procedural Activitiescited by 51
  11. 2026World Action Models: The Next Frontier in Embodied AIcited by 50
  12. 2024Aria Everyday Activities Datasetcited by 39
38 more
  1. 2024PREGO: online mistake detection in PRocedural EGOcentric videoscited by 39
  2. 2024EgoExo-Fitness: Towards Egocentric and Exocentric Full-Body Action Understandingcited by 36
  3. 2025PAI-Bench: A Comprehensive Benchmark For Physical AIcited by 34
  4. 2025World Models for Learning Dexterous Hand-Object Interactions from Human Videoscited by 31
  5. 2024ObjectRelator: Enabling Cross-View Object Relation Understanding Across Ego-Centric and Exo-Centric Perspectivescited by 29
  6. 2025Proactive Assistant Dialogue Generation from Streaming Egocentric Videoscited by 28
  7. 2024EgoGen: An Egocentric Synthetic Data Generatorcited by 27
  8. 2024Differentiable Task Graph Learning: Procedural Activity Representation and Online Mistake Detection from Egocentric Videoscited by 26
  9. 2025In-N-On: Scaling Egocentric Manipulation with in-the-wild and on-task Datacited by 26
  10. 2025Robix: A Unified Model for Robot Interaction, Reasoning and Planningcited by 23
  11. 2026A Systematic Study of Data Modalities and Strategies for Co-training Large Behavior Models for Robot Manipulationcited by 22
  12. 2025EgoBlind: Towards Egocentric Visual Assistance for the Blindcited by 21
  13. 2024What to Say and When to Say it: Live Fitness Coaching as a Testbed for Situated Interactioncited by 18
  14. 2026LAP: Language-Action Pre-Training Enables Zero-shot Cross-Embodiment Transfercited by 18
  15. 2025The Invisible EgoHand: 3D Hand Forecasting through EgoBody Pose Estimationcited by 16
  16. 2026HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videoscited by 16
  17. 2024Gazing Into Missteps: Leveraging Eye-Gaze for Unsupervised Mistake Detection in Egocentric Videos of Skilled Human Activitiescited by 15
  18. 2025OpenEgo: A Large-Scale Multimodal Egocentric Dataset for Dexterous Manipulationcited by 13
  19. 2025Co-Evolving Latent Action World Modelscited by 13
  20. 2025Modeling Multiple Normal Action Representations for Error Detection in Procedural Taskscited by 12
  21. 2024PrISM-Observer: Intervention Agent to Help Users Perform Everyday Procedures Sensed using a Smartwatchcited by 11
  22. 2024EgoCast: Forecasting Egocentric Human Pose in the Wildcited by 11
  23. 2025egoPPG: Heart Rate Estimation from Eye-Tracking Cameras in Egocentric Systems to Benefit Downstream Vision Taskscited by 11
  24. 2025Generating 6DoF Object Manipulation Trajectories from Action Description in Egocentric Visioncited by 11
  25. 2025Developing Vision-Language-Action Model from Egocentric Videoscited by 11
  26. 2026EgoSound: Benchmarking Sound Understanding in Egocentric Videoscited by 11
  27. 2024TI-PREGO: Chain of Thought and In-Context Learning for Online Mistake Detection in PRocedural EGOcentric Videoscited by 10
  28. 2025METIS: Multi-Source Egocentric Training for Integrated Dexterous Vision-Language-Action Modelcited by 10
  29. 2025When Robots Should Say "I Don't Know": Benchmarking Abstention in Embodied Question Answeringcited by 10
  30. 2026PhyCritic: Multimodal Critic Models for Physical AIcited by 10
  31. 2025MEgoHand: Multimodal Egocentric Hand-Object Interaction Motion Generationcited by 9
  32. 2025EgoM2P: Egocentric Multimodal Multitask Pretrainingcited by 9
  33. 2024Autonomous Workflow for Multimodal Fine-Grained Training Assistants Towards Mixed Realitycited by 8
  34. 2024SIGMA: An Open-Source Interactive System for Mixed-Reality Task Assistance Researchcited by 8
  35. 2024Is 'Right' Right? Enhancing Object Orientation Understanding in Multimodal Large Language Models through Egocentric Instruction Tuningcited by 8
  36. 2025EgoMe: A New Dataset and Challenge for Following Me via Egocentric View in Real Worldcited by 8
  37. 2025Perceiving and Acting in First-Person: A Dataset and Benchmark for Egocentric Human-Object-Human Interactionscited by 8
  38. 2025Flowing from Reasoning to Motion: Learning 3D Hand Trajectory Prediction from Egocentric Human Interaction Videoscited by 8

50 most cited of 166 All 166 (JSON)