mlsanityData to 7 Oct 2026

Action Scene Graphs for Long-Form Understanding of Egocentric Videos

arXiv 2312.03391 ↗First posted 6 Dec 2023cs.CVcited by 51

Not on the map. Why?

Abstract (arXiv)

We present Egocentric Action Scene Graphs (EASGs), a new representation for long-form understanding of egocentric videos. EASGs extend standard manually-annotated representations of egocentric videos, such as verb-noun action labels, by providing a temporally evolving graph-based description of the actions performed by the camera wearer, including interacted objects, their relationships, and how actions unfold in time. Through a novel annotation procedure, we extend the Ego4D dataset by adding manually labeled Egocentric Action Scene Graphs offering a rich set of annotations designed for long-from egocentric video understanding. We hence define the EASG generation task and provide a baseline approach, establishing preliminary benchmarks. Experiments on two downstream tasks, egocentric action anticipation and egocentric activity summarization, highlight the effectiveness of EASGs for long-form egocentric video understanding. We will release the dataset and the code to replicate experiments and annotations.

Key facts. Cited by 51 papers in the map (data through 7 Oct 2026). Builds on 39 of its 55 references that are in the map. First posted on arXiv 6 Dec 2023.

Builds on

39 of 55 references.

References · 39

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  4. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  5. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  6. 2018SlowFast Networks for Video Recognitioncited by 1,862
  7. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  8. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  9. 2020Rescaling Egocentric Visioncited by 529
  10. 2018Image Generation from Scene Graphscited by 324
  11. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  12. 2018Learning to Compose Dynamic Tree Structures for Visual Contextscited by 258
27 more
  1. 2018Auto-Encoding Scene Graphs for Image Captioningcited by 254
  2. 2019Action Genome: Actions as Composition of Spatio-temporal Scene Graphscited by 249
  3. 2018Videos as Space-Time Region Graphscited by 221
  4. 2023Large Language Models as General Pattern Machinescited by 196
  5. 2021Towards Long-Form Video Understandingcited by 146
  6. 2022EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relationscited by 139
  7. 2022Panoptic Scene Graph Generationcited by 114
  8. 2020EGO-TOPO: Environment Affordances from Egocentric Videocited by 111
  9. 2021SGTR: End-to-end Scene Graph Generation with Transformercited by 102
  10. 2023An Outlook into the Future of Egocentric Visioncited by 84
  11. 2019Learning Canonical Representations for Scene Graph to Image Generationcited by 60
  12. 2021Home Action Genome: Cooperative Compositional Action Understandingcited by 52
  13. 2023Panoptic Video Scene Graph Generationcited by 46
  14. 2021Unified Graph Structured Models for Video Understandingcited by 38
  15. 2023Unbiased Scene Graph Generation in Videoscited by 38
  16. 2021Predicting the Future from First Person (Egocentric) Vision: A Surveycited by 36
  17. 2019Adaptive Graph Representation Learning for Video Person Re-identificationcited by 34
  18. 2021In Defense of Scene Graphs for Image Captioningcited by 31
  19. 2021Exploiting Long-Term Dependencies for Generating Dynamic Scene Graphscited by 30
  20. 2017Temporal Dynamic Graph LSTM for Action-driven Video Object Detectioncited by 23
  21. 2021ReFormer: The Relational Transformer for Image Captioningcited by 23
  22. 2021Exploiting Scene Graphs for Human-Object Interaction Detectioncited by 22
  23. 2021roadscene2vec: A Tool for Extracting and Embedding Road Scene-Graphscited by 15
  24. 2023Constructing Holistic Spatio-Temporal Scene Graph for Video Semantic Role Labelingcited by 15
  25. 2022Intel Labs at Ego4D Challenge 2022: A Better Baseline for Audio-Visual Diarizationcited by 9
  26. 2023Explainable Action Prediction through Self-Supervision on Scene Graphscited by 9
  27. 2023STHG: Spatial-Temporal Heterogeneous Graph Learning for Advanced Audio-Visual Diarizationcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 26; 2026: 20.

51 papers.

Sort:

By area:

  1. 2025Ego-R1: Chain-of-Tool-Thought for Ultra-Long Egocentric Video Reasoningcited by 67
  2. 2024Grounded Multi-Hop VideoQA in Long-Form Egocentric Videoscited by 43
  3. 2024AMEGO: Active Memory from long EGOcentric videoscited by 35
  4. 2025Challenges and Trends in Egocentric Vision: A Surveycited by 20
  5. 2025Building a Mind Palace: Structuring Environment-Grounded Semantic Graphs for Effective Long Video Analysis with LLMscited by 19
  6. 2025VideoMultiAgents: A Multi-Agent Framework for Video Question Answeringcited by 16
  7. 2025X-LeBench: A Benchmark for Extremely Long Egocentric Video Understandingcited by 14
  8. 2024FineBio: A Fine-Grained Video Dataset of Biological Experiments with Hierarchical Annotationcited by 11
  9. 2025Universal Scene Graph Generationcited by 11
  10. 2025Generating 6DoF Object Manipulation Trajectories from Action Description in Egocentric Visioncited by 11
  11. 2024Motion-aware Contrastive Learning for Temporal Panoptic Scene Graph Generationcited by 9
  12. 2025LLM Meets Scene Graph: Can Large Language Models Understand and Generate Scene Graphs? A Benchmark and Empirical Studycited by 9
38 more
  1. 2024ActionVOS: Actions as Prompts for Video Object Segmentationcited by 8
  2. 2025Learning 4D Panoptic Scene Graph Generation from Rich 2D Visual Scenecited by 6
  3. 2025From Pixels to Graphs: using Scene and Knowledge Graphs for HD-EPIC VQA Challengecited by 6
  4. 2025EASG-Bench: Video Q&A Benchmark with Egocentric Action Scene Graphscited by 5
  5. 2025Building Egocentric Procedural AI Assistant: Methods, Benchmarks, and Challengescited by 5
  6. 2025DyGEnc: Encoding a Sequence of Textual Scene Graphs to Reason and Answer Questions in Dynamic Scenescited by 4
  7. 2025CAT-SG: A Large Dynamic Scene Graph Dataset for Fine-Grained Understanding of Cataract Surgerycited by 4
  8. 2025From Vision To Language through Graph of Events in Space and Time: An Explainable Self-supervised Approachcited by 4
  9. 2025Generating Directed Graphs with Dual Attention and Asymmetric Encodingcited by 3
  10. 2025Decoding the Surgical Scene: A Scoping Review of Scene Graphs in Surgerycited by 3
  11. 2026Synthetic Visual Genome 2: Extracting Large-scale Spatio-Temporal Scene Graphs from Videoscited by 3
  12. 2026EgoGraph: Temporal Knowledge Graph for Egocentric Video Understandingcited by 3
  13. 2026FocusGraph: Graph-Structured Frame Selection for Embodied Long Video Question Answeringcited by 3
  14. 2026EgoSelf: From Memory to Personalized Egocentric Assistantcited by 2
  15. 2025Stitch-a-Demo: Video Demonstrations from Multistep Descriptionscited by 1
  16. 2025METOR: A Unified Framework for Mutual Enhancement of Objects and Relationships in Open-vocabulary Video Visual Relationship Detectioncited by 1
  17. 2025DiscoSG: Towards Discourse-Level Text Scene Graph Parsing through Iterative Graph Refinementcited by 1
  18. 2025UNO: Unifying One-stage Video Scene Graph Generation via Object-Centric Visual Representation Learningcited by 1
  19. 2025EventFormer: A Node-graph Hierarchical Attention Transformer for Action-centric Video Event Predictioncited by 1
  20. 2025RocSync: Millisecond-Accurate Temporal Synchronization for Heterogeneous Camera Systemscited by 1
  21. 2026Revisiting Weakly-Supervised Video Scene Graph Generation via Pair Affinity Learningcited by 1
  22. 2026Graph it first! Enabling Reasoning on Long-form Egocentric Videos through Scene Graphscited by 1
  23. 2025Graph-Based Multimodal and Multi-view Alignment for Keystep Recognitioncited by 0
  24. 2025REVEAL: Relation-based Video Representation Learning for Video-Question-Answeringcited by 0
  25. 2025Keystep Recognition using Graph Neural Networkscited by 0
  26. 2026CSA-Graphs: A Privacy-Preserving Structural Dataset for Child Sexual Abuse Researchcited by 0
  27. 2026EgoEverything: A Benchmark for Human Behavior Inspired Long Context Egocentric Video Understanding in AR Environmentcited by 0
  28. 2026Frequency-guided Multi-level Reasoning for Scene Graph Generation in Videocited by 0
  29. 2026VLESA: Vision-Language Embodied Safety Agent for Human Activity Monitoringcited by 0
  30. 2026From Frames to Temporal Graphs: In-Context Egocentric Action Recognition with Vision-Language Modelscited by 0
  31. 2026GRACE: Boosting Video MLLMs with Grounded Action-Centric Evidence for Viewer Sentiment Predictioncited by 0
  32. 2026Learning to Evolve Scenes: Reasoning about Human Activities with Scene Graphscited by 0
  33. 2026Beyond Pairwise Relations: Dynamic Manipulation Hypergraphs for Vision-Based Human Activity Recognitioncited by 0
  34. 2026GESTO: Human-Centric Spatio-Temporal Memory for Reasoning in Dynamic Scenescited by 0
  35. 2026SAGE-OR: Semi-supervised Adaptive Scene Graph Generation for Operating Roomscited by 0
  36. 2026G3Ego: Gaze-Guided Graphs for Egocentric Action Understandingcited by 0
  37. 2026Reliable Egocentric Action Anticipation via Temporal Reliability Suppression and Compositional Graph Decodingcited by 0
  38. 2026HAP: A Hand-Driven Active Perception Framework for Egocentric Head Motion Predictioncited by 0

50 most cited of 51 All 51 (JSON)