mlsanityData to 14 Oct 2026

HAtt-Flow: Hierarchical Attention-Flow Mechanism for Group Activity Scene Graph Generation in Videos

arXiv 2312.07740 ↗First posted 28 Nov 2023cs.CVcited by 4

Not on the map. Why?

Abstract (arXiv)

Group Activity Scene Graph (GASG) generation is a challenging task in computer vision, aiming to anticipate and describe relationships between subjects and objects in video sequences. Traditional Video Scene Graph Generation (VidSGG) methods focus on retrospective analysis, limiting their predictive capabilities. To enrich the scene understanding capabilities, we introduced a GASG dataset extending the JRDB dataset with nuanced annotations involving textitAppearance, Interaction, Position, Relationship, and Situation attributes. This work also introduces an innovative approach, Hierarchical Attention-Flow (HAtt-Flow) Mechanism, rooted in flow network theory to enhance GASG performance. Flow-Attention incorporates flow conservation principles, fostering competition for sources and allocation for sinks, effectively preventing the generation of trivial attention. Our proposed approach offers a unique perspective on attention mechanisms, where conventional "values" and "keys" are transformed into sources and sinks, respectively, creating a novel framework for attention-based models. Through extensive experiments, we demonstrate the effectiveness of our Hatt-Flow model and the superiority of our proposed Flow-Attention mechanism. This work represents a significant advancement in predictive video scene understanding, providing valuable insights and techniques for applications that require real-time relationship prediction in video data.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 52 of its 81 references that are in the map. First posted on arXiv 28 Nov 2023.

Builds on

52 of 81 references.

References · 52

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2017Scene Graph Generation by Iterative Message Passingcited by 462
  3. 2017Neural Motifs: Scene Graph Parsing with Global Contextcited by 414
  4. 2020Unbiased Scene Graph Generation from Biased Trainingcited by 356
  5. 2016Visual Relationship Detection with Language Priorscited by 334
  6. 2018Graph R-CNN for Scene Graph Generationcited by 298
  7. 2018Learning to Compose Dynamic Tree Structures for Visual Contextscited by 259
  8. 2017Learning to Detect Human-Object Interactionscited by 240
  9. 2017Detecting and Recognizing Human-Object Interactionscited by 199
  10. 2015Visual Semantic Role Labelingcited by 190
  11. 2021HOTR: End-to-End Human-Object Interaction Detection with Transformerscited by 183
  12. 2020GPS-Net: Graph Property Sensing Network for Scene Graph Generationcited by 166
38 more
  1. 2019Knowledge-Embedded Routing Network for Scene Graph Generationcited by 164
  2. 2021QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Informationcited by 147
  3. 2021End-to-End Human Object Interaction Detection with HOI Transformercited by 135
  4. 2020Visual Compositional Learning for Human-Object Interaction Detectioncited by 130
  5. 2020DRG: Dual Relation Graph for Human-Object Interaction Detectioncited by 117
  6. 2022Panoptic Scene Graph Generationcited by 114
  7. 2017Visual Translation Embedding Network for Visual Relation Detectioncited by 112
  8. 2020Learning Human-Object Interaction Detection using Interaction Pointscited by 108
  9. 2021Energy-Based Learning for Scene Graph Generationcited by 104
  10. 2021Mining the Benefits of Two-stage and One-stage HOI Detectioncited by 103
  11. 2019Scene Graph Generation with External Knowledge and Image Reconstructioncited by 102
  12. 2017Detecting Visual Relationships with Deep Relational Networkscited by 97
  13. 2020Detailed 2D-3D Joint Representation for Human-Object Interactioncited by 93
  14. 2021Efficient Two-Stage Detection of Human-Object Interactions with a Novel Unary-Pairwise Transformercited by 91
  15. 2016Hierarchical Deep Temporal Models for Group Activity Recognitioncited by 89
  16. 2020Bridging Knowledge Graphs to Generate Scene Graphscited by 86
  17. 2020Actor-Transformers for Group Activity Recognitioncited by 84
  18. 2021Affordance Transfer Learning for Human-Object Interaction Detectioncited by 79
  19. 2018Factorizable Net: An Efficient Subgraph-based Framework for Scene Graph Generationcited by 78
  20. 2018Attentive Relational Networks for Mapping Images to Scene Graphscited by 73
  21. 2021Learning of Visual Relations: The Devil is in the Tailscited by 65
  22. 2021Learning to Generate Scene Graph from Natural Language Supervisioncited by 64
  23. 2019Deep Contextual Attention for Human-Object Interaction Detectioncited by 54
  24. 2021Target Adaptive Context Aggregation for Video Scene Graph Generationcited by 52
  25. 2020Learning Visual Commonsense for Robust Scene Graph Generationcited by 47
  26. 2016Video Captioning with Transferred Semantic Attributescited by 46
  27. 2016Social Scene Understanding: End-to-End Multi-Person Action Localization and Collective Activity Recognitioncited by 43
  28. 2020Cascaded Human-Object Interaction Recognitioncited by 40
  29. 2018Hierarchical Long Short-Term Concurrent Memory for Human Interaction Recognitioncited by 36
  30. 2015Joint Inference of Groups, Events and Human Roles in Aerial Videoscited by 30
  31. 2021Linguistic Structures as Weak Supervision for Visual Scene Graph Generationcited by 29
  32. 2015Structure Inference Machines: Recurrent Neural Networks for Analyzing Relations in Group Activity Recognitioncited by 27
  33. 2022Dual-AI: Dual-path Actor Interaction Learning for Group Activity Recognitioncited by 27
  34. 2021Segmentation-grounded Scene Graph Generationcited by 25
  35. 2023Type-to-Track: Retrieve Any Object via Prompt-based Trackingcited by 24
  36. 2022Hunting Group Clues with Transformers for Social Group Activity Recognitioncited by 22
  37. 2018Detecting Visual Relationships Using Box Attentioncited by 16
  38. 2023SPARTAN: Self-supervised Spatiotemporal Transformers Approach to Group Activity Recognitioncited by 14

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 4.

4 papers.

  1. 2024CYCLO: Cyclic Graph Transformer Approach to Multi-Object Relationship Modeling in Aerial Videoscited by 11
  2. 2024FLAASH: Flow-Attention Adaptive Semantic Hierarchical Fusion for Multi-Modal Tobacco Content Analysiscited by 2
  3. 2024Public Health Advocacy Dataset: A Dataset of Tobacco Usage Videos from Social Mediacited by 2
  4. 2024LiGAR: LiDAR-Guided Hierarchical Transformer for Multi-Modal Group Activity Recognitioncited by 1