mlsanityData to 7 Oct 2026

Action Genome: Actions as Composition of Spatio-temporal Scene Graphs

arXiv 1912.06992 ↗First posted 15 Dec 2019cs.CVcited by 249

Not on the map. Why?

Cited by 26 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Action recognition has typically treated actions and activities as monolithic events that occur in videos. However, there is evidence from Cognitive Science and Neuroscience that people actively encode activities into consistent hierarchical part structures. However in Computer Vision, few explorations on representations encoding event partonomies have been made. Inspired by evidence that the prototypical unit of an event is an action-object interaction, we introduce Action Genome, a representation that decomposes actions into spatio-temporal scene graphs. Action Genome captures changes between objects and their pairwise relationships while an action occurs. It contains 10K videos with 0.4M objects and 1.7M visual relationships annotated. With Action Genome, we extend an existing action recognition model by incorporating scene graphs as spatio-temporal feature banks to achieve better performance on the Charades dataset. Next, by decomposing and learning the temporal changes in visual relationships that result in an action, we demonstrate the utility of a hierarchical event decomposition by enabling few-shot action recognition, achieving 42.7% mAP using as few as 10 examples. Finally, we benchmark existing scene graph models on the new task of spatio-temporal scene graph prediction.

Key facts. Cited by 249 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 Dec 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 30; 2022: 40; 2023: 55; 2024: 37; 2025: 55; 2026: 32.

249 papers.

Sort:

By area:

  1. 2021Grounded Language-Image Pre-trainingcited by 1,147
  2. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionalitycited by 238
  3. 2024STAR: A Benchmark for Situated Reasoning in Real-World Videoscited by 213
  4. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 177
  5. 2024Video-of-Thought: Step-by-Step Video Reasoning from Perception to Cognitioncited by 151
  6. 2021Towards Long-Form Video Understandingcited by 146
  7. 2022EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relationscited by 139
  8. 2021TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?cited by 127
  9. 2021A Comprehensive Survey of Scene Graphs: Generation and Applicationcited by 125
  10. 2021AGQA: A Benchmark for Compositional Spatio-Temporal Reasoningcited by 120
  11. 2022CLEVR-Math: A Dataset for Compositional Language, Visual and Mathematical Reasoningcited by 119
  12. 2022EgoTaskQA: Understanding Human Tasks in Egocentric Videoscited by 112
38 more
  1. 2021Spatial-Temporal Transformer for Dynamic Scene Graph Generationcited by 95
  2. 2022RelTR: Relation Transformer for Scene Graph Generationcited by 95
  3. 2023ReVersion: Diffusion-Based Relation Inversion from Imagescited by 82
  4. 2022Scene Graph Generation: A Comprehensive Surveycited by 73
  5. 2023Dense and Aligned Captions (DAC) Promote Compositional Reasoning in VL Modelscited by 72
  6. 2022Teaching Structured Vision&Language Concepts to Vision&Language Modelscited by 71
  7. 2021Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoningcited by 68
  8. 2022Relationformer: A Unified Framework for Image-to-Graph Generationcited by 67
  9. 2021Learning to Generate Scene Graph from Natural Language Supervisioncited by 64
  10. 2022RLIP: Relational Language-Image Pre-training for Human-Object Interaction Detectioncited by 54
  11. 2023COLA: A Benchmark for Compositional Text-to-image Retrievalcited by 53
  12. 2021Home Action Genome: Cooperative Compositional Action Understandingcited by 52
  13. 2021Target Adaptive Context Aggregation for Video Scene Graph Generationcited by 51
  14. 2023Equivariant Similarity for Vision-Language Foundation Modelscited by 51
  15. 2023Action Scene Graphs for Long-Form Understanding of Egocentric Videoscited by 51
  16. 20224D-OR: Semantic Scene Graphs for OR Domain Modelingcited by 49
  17. 2023Panoptic Video Scene Graph Generationcited by 46
  18. 2023HiCLIP: Contrastive Language-Image Pretraining with Hierarchy-aware Attentioncited by 44
  19. 2023RLIPv2: Fast Scaling of Relational Language-Image Pre-trainingcited by 44
  20. 2023SceneGenie: Scene Graph Guided Diffusion Models for Image Synthesiscited by 41
  21. 2023Benchmarks for Automated Commonsense Reasoning: A Surveycited by 40
  22. 2022Continuous Scene Representations for Embodied AIcited by 39
  23. 2023Incorporating Structured Representations into Pretrained Vision & Language Models Using Scene Graphscited by 39
  24. 2021Unified Graph Structured Models for Video Understandingcited by 38
  25. 2023Unbiased Scene Graph Generation in Videoscited by 38
  26. 2022Latent Graph Representations for Critical View of Safety Assessmentcited by 37
  27. 2023Dysen-VDM: Empowering Dynamics-aware Text-to-Video Diffusion with LLMscited by 35
  28. 2024AMEGO: Active Memory from long EGOcentric videoscited by 35
  29. 2021Deep Learning-based Action Detection in Untrimmed Videos: A Surveycited by 32
  30. 2023Contrastive Video Question Answering via Video Graph Transformercited by 32
  31. 2024Enhancing Video-Language Representations with Structural Spatio-Temporal Alignmentcited by 32
  32. 2024Learning Action and Reasoning-Centric Image Editing from Videos and Simulationscited by 32
  33. 2021TCGL: Temporal Contrastive Graph for Self-supervised Video Representation Learningcited by 31
  34. 2022Do Vision-Language Pretrained Models Learn Composable Primitive Concepts?cited by 31
  35. 2021Exploiting Long-Term Dependencies for Generating Dynamic Scene Graphscited by 30
  36. 2021Visual Scene Graphs for Audio Source Separationcited by 29
  37. 2022(2.5+1)D Spatio-Temporal Scene Graphs for Video Question Answeringcited by 29
  38. 2021Coarse-Fine Networks for Temporal Activity Detection in Videoscited by 28

50 most cited of 249 All 249 (JSON)