mlsanityData to 7 Oct 2026

EgoTaskQA: Understanding Human Tasks in Egocentric Videos

arXiv 2210.03929 ↗First posted 8 Oct 2022cs.CV cs.AI cs.CLcited by 112

Not on the map. Why?

Cited by 35 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Understanding human tasks through video observations is an essential capability of intelligent agents. The challenges of such capability lie in the difficulty of generating a detailed understanding of situated actions, their effects on object states (i.e., state changes), and their causal dependencies. These challenges are further aggravated by the natural parallelism from multi-tasking and partial observations in multi-agent collaboration. Most prior works leverage action localization or future prediction as an indirect metric for evaluating such task understanding from videos. To make a direct evaluation, we introduce the EgoTaskQA benchmark that provides a single home for the crucial dimensions of task understanding through question-answering on real-world egocentric videos. We meticulously design questions that target the understanding of (1) action dependencies and effects, (2) intents and goals, and (3) agents' beliefs about others. These questions are divided into four types, including descriptive (what status?), predictive (what will?), explanatory (what caused?), and counterfactual (what if?) to provide diagnostic analyses on spatial, temporal, and causal understandings of goal-oriented tasks. We evaluate state-of-the-art video reasoning models on our benchmark and show their significant gaps between humans in understanding complex goal-oriented egocentric videos. We hope this effort will drive the vision community to move onward with goal-oriented video understanding and reasoning.

Key facts. Cited by 112 papers in the map (data through 7 Oct 2026). Builds on 69 of its 147 references that are in the map. First posted on arXiv 8 Oct 2022.

Builds on

69 of 147 references.

References · 69

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  3. 2015VQA: Visual Question Answeringcited by 2,489
  4. 2018SlowFast Networks for Video Recognitioncited by 1,862
  5. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  6. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  7. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  8. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  9. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
  10. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  11. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  12. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
38 more
  1. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  2. 2020Rescaling Egocentric Visioncited by 529
  3. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  4. 2019CLEVRER: CoLlision Events for Video REpresentation and Reasoningcited by 475
  5. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
  6. 2017TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answeringcited by 378
  7. 2016Generating Videos with Scene Dynamicscited by 376
  8. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  9. 2019COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysiscited by 316
  10. 2015MovieQA: Understanding Stories in Movies through Question-Answeringcited by 304
  11. 2019Action Genome: Actions as Composition of Spatio-temporal Scene Graphscited by 249
  12. 2020Just Ask: Learning to Answer Questions from Millions of Narrated Videoscited by 248
  13. 2017First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotationscited by 229
  14. 2019RAVEN: A Dataset for Relational and Analogical Visual rEasoNingcited by 225
  15. 2022Omnivore: A Single Model for Many Visual Modalitiescited by 214
  16. 2018WikiHow: A Large Scale Text Summarization Datasetcited by 203
  17. 2021Anticipative Video Transformercited by 197
  18. 2021TEACh: Task-driven Embodied Agents that Chatcited by 194
  19. 2022MeMViT: Memory-Augmented Multiscale Vision Transformer for Efficient Long-Term Video Recognitioncited by 189
  20. 2018Learning Human-Object Interactions by Graph Parsing Neural Networkscited by 176
  21. 2015Action-Conditional Video Prediction using Deep Networks in Atari Gamescited by 171
  22. 2018Measuring abstract reasoning in neural networkscited by 165
  23. 2020Hierarchical Conditional Relation Networks for Video Question Answeringcited by 163
  24. 2020In Defense of Grid Features for Visual Question Answeringcited by 155
  25. 2019TVQA+: Spatio-Temporal Grounding for Video Question Answeringcited by 153
  26. 2019Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answeringcited by 146
  27. 2018Grounded Human-Object Interaction Hotspots from Videocited by 128
  28. 2018Charades-Ego: A Large-Scale Dataset of Paired Third and First Person Videoscited by 127
  29. 2021AGQA: A Benchmark for Compositional Spatio-Temporal Reasoningcited by 120
  30. 2020Rolling-Unrolling LSTMs for Action Anticipation from First-Person Videocited by 119
  31. 2015Unsupervised Learning from Narrated Instruction Videoscited by 116
  32. 2020EGO-TOPO: Environment Affordances from Egocentric Videocited by 111
  33. 2019Procedure Planning in Instructional Videoscited by 92
  34. 2022Learning To Recognize Procedural Activities with Distant Supervisioncited by 85
  35. 2019You2Me: Inferring Body Pose in Egocentric Video via First and Second Person Interactionscited by 82
  36. 2017DeepStory: Video Story QA by Deep Embedded Memory Networkscited by 67
  37. 2020Bongard-LOGO: A New Benchmark for Human-Level Concept Learning and Reasoningcited by 63
  38. 2017Future Person Localization in First-Person Videoscited by 57

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 8; 2024: 23; 2025: 48; 2026: 32.

112 papers.

Sort:

By area:

  1. 2022SQA3D: Situated Question Answering in 3D Scenescited by 303
  2. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  3. 2024VideoAgent: A Memory-augmented Multimodal Agent for Video Understandingcited by 224
  4. 2023EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbonecited by 130
  5. 2024Video-CCAM: Enhancing Video-Language Understanding with Causal Cross-Attention Masks for Short and Long Videoscited by 95
  6. 2024EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real Worldcited by 93
  7. 2025EgoLife: Towards Egocentric Life Assistantcited by 86
  8. 2023An Outlook into the Future of Egocentric Visioncited by 84
  9. 2024Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-trainingcited by 82
  10. 2024MME-Survey: A Comprehensive Survey on Evaluation of Multimodal LLMscited by 72
  11. 2023EgoPlan-Bench: Benchmarking Multimodal Large Language Models for Human-Level Planningcited by 66
  12. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 66
38 more
  1. 2025Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Modelscited by 62
  2. 2024MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Modelscited by 61
  3. 2023Procedure-Aware Pretraining for Instructional Video Understandingcited by 48
  4. 2025EgoTextVQA: Towards Egocentric Scene-Text Aware Video Question Answeringcited by 43
  5. 2025Omnia de EgoTempo: Benchmarking Temporal Understanding of Multi-Modal LLMs in Egocentric Videoscited by 41
  6. 2023Memory-and-Anticipation Transformer for Online Action Understandingcited by 39
  7. 2024EgoPlan-Bench2: A Benchmark for Multimodal Large Language Model Planning in Real-World Scenarioscited by 37
  8. 2025Scaling Video-Language Models to 10K Frames via Hierarchical Differential Distillationcited by 33
  9. 2024Situational Awareness Matters in 3D Vision Language Reasoningcited by 32
  10. 2024EgoLM: Multi-Modal Language Model of Egocentric Motionscited by 28
  11. 2025InteractiveOmni: A Unified Omni-modal Model for Audio-Visual Multi-turn Dialoguecited by 27
  12. 2024ObjectRelator: Enabling Cross-View Object Relation Understanding Across Ego-Centric and Exo-Centric Perspectivescited by 26
  13. 2024PVC: Progressive Visual Token Compression for Unified Image and Video Processing in Large Vision-Language Modelscited by 25
  14. 2025Exo2Ego: Exocentric Knowledge Guided MLLM for Egocentric Video Understandingcited by 25
  15. 2024Satori: Towards Proactive AR Assistant with Belief-Desire-Intention User Modelingcited by 24
  16. 2024Embodied VideoAgent: Persistent Memory from Egocentric Videos and Embodied Sensors Enables Dynamic Scene Understandingcited by 24
  17. 2025FAVOR-Bench: A Comprehensive Benchmark for Fine-Grained Video Motion Understandingcited by 24
  18. 2025Robix: A Unified Model for Robot Interaction, Reasoning and Planningcited by 23
  19. 2025EgoExoBench: A Benchmark for First- and Third-person View Video Understanding in MLLMscited by 22
  20. 2025Uni-MoE-2.0-Omni: Scaling Language-Centric Omnimodal Large Model with Advanced MoE, Training and Datacited by 21
  21. 2025Challenges and Trends in Egocentric Vision: A Surveycited by 20
  22. 2025M2-omni: Advancing Omni-MLLM for Comprehensive Modality Support with Competitive Performancecited by 19
  23. 2025EgoBlind: Towards Egocentric Visual Assistance for the Blindcited by 19
  24. 2025EgoCross: Benchmarking Multimodal Large Language Models for Cross-Domain Egocentric Video Question Answeringcited by 19
  25. 2025Ego4o: Egocentric Human Motion Capture and Understanding from Multi-Modal Inputcited by 17
  26. 2025EgoNight: Towards Egocentric Vision Understanding at Night with a Challenging Benchmarkcited by 16
  27. 2025X-LeBench: A Benchmark for Extremely Long Egocentric Video Understandingcited by 14
  28. 2025SAVVY: Spatial Awareness via Audio-Visual LLMs through Seeing and Hearingcited by 14
  29. 2025CLiViS: Unleashing Cognitive Map through Linguistic-Visual Synergy for Embodied Visual Reasoningcited by 13
  30. 2025Mono-InternVL-1.5: Towards Cheaper and Faster Monolithic Multimodal Large Language Modelscited by 13
  31. 2025LongViTU: Instruction Tuning for Long-Form Video Understandingcited by 12
  32. 2024Long Story Short: Story-level Video Understanding from 20K Short Filmscited by 11
  33. 2024Grounding 3D Scene Affordance From Egocentric Interactionscited by 11
  34. 2024ET-Plan-Bench: Embodied Task-level Planning Benchmark Towards Spatial-Temporal Cognition with Foundation Modelscited by 11
  35. 2023STRAP: Structured Object Affordance Segmentation with Point Supervisioncited by 10
  36. 2024HoVLE: Unleashing the Power of Monolithic Vision-Language Models with Holistic Vision-Language Embeddingcited by 10
  37. 2025EgoToM: Benchmarking Theory of Mind Reasoning from Egocentric Videoscited by 10
  38. 2024Video Question Answering for People with Visual Impairments Using an Egocentric 360-Degree Cameracited by 9

50 most cited of 112 All 112 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026EgoIntent: A Pre-Outcome Micro-Step Benchmark for Understanding What, Why, and Next
  2. 2023EgoTV: Egocentric Task Verification from Natural Language Task Descriptions
  3. 2025EgoThinker: Unveiling Egocentric Reasoning with Spatio-Temporal CoT
  4. 2026MA-EgoQA: Question Answering over Egocentric Videos from Multiple Embodied Agents
  5. 2026EgoExo-Next:Benchmarking Vision-Language Models on Visual-Option Next-State and Cross-View Reasoning
  6. 2025In the Eye of MLLM: Benchmarking Egocentric Video Intent Understanding with Gaze-Guided Prompting
  7. 2025ENACT: Evaluating Embodied Cognition with World Modeling of Egocentric Interaction
  8. 2024Simultaneous Localization and Affordance Prediction of Tasks from Egocentric Video