mlsanityData to 14 Oct 2026

HyperGLM: HyperGraph for Video Scene Graph Generation and Anticipation

arXiv 2411.18042 ↗First posted 27 Nov 2024cs.CVcited by 19

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Multimodal LLMs have advanced vision-language tasks but still struggle with understanding video scenes. To bridge this gap, Video Scene Graph Generation (VidSGG) has emerged to capture multi-object relationships across video frames. However, prior methods rely on pairwise connections, limiting their ability to handle complex multi-object interactions and reasoning. To this end, we propose Multimodal LLMs on a Scene HyperGraph (HyperGLM), promoting reasoning about multi-way interactions and higher-order relationships. Our approach uniquely integrates entity scene graphs, which capture spatial relationships between objects, with a procedural graph that models their causal transitions, forming a unified HyperGraph. Significantly, HyperGLM enables reasoning by injecting this unified HyperGraph into LLMs. Additionally, we introduce a new Video Scene Graph Reasoning (VSGR) dataset featuring 1.9M frames from third-person, egocentric, and drone views and supports five tasks: Scene Graph Generation, Scene Graph Anticipation, Video Question Answering, Video Captioning, and Relation Reasoning. Empirically, HyperGLM consistently outperforms state-of-the-art methods across five tasks, effectively modeling and reasoning complex relationships in diverse video scenes.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 42 of its 56 references that are in the map. First posted on arXiv 27 Nov 2024.

Builds on

42 of 56 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Models1 Apr

Other references · 41

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2023Mistral 7Bcited by 3,164
  6. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  7. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  8. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 546
  9. 2023MovieChat: From Dense Token to Sparse Memory for Long Video Understandingcited by 542
  10. 2023Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understandingcited by 371
  11. 2019Action Genome: Actions as Composition of Spatio-temporal Scene Graphscited by 250
  12. 2024MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understandingcited by 229
29 more
  1. 2019Hypergraph Convolution and Hypergraph Attentioncited by 228
  2. 2024VideoLLM-online: Online Video Large Language Model for Streaming Videocited by 225
  3. 2024Video-of-Thought: Step-by-Step Video Reasoning from Perception to Cognitioncited by 153
  4. 2022End-to-end Generative Pretraining for Multimodal Video Captioningcited by 116
  5. 2022Panoptic Scene Graph Generationcited by 114
  6. 2021SGTR: End-to-end Scene Graph Generation with Transformercited by 103
  7. 2021Spatial-Temporal Transformer for Dynamic Scene Graph Generationcited by 96
  8. 2022RelTR: Relation Transformer for Scene Graph Generationcited by 96
  9. 2022Towards Open-vocabulary Scene Graph Generation with Prompt-based Finetuningcited by 54
  10. 2021Target Adaptive Context Aggregation for Video Scene Graph Generationcited by 52
  11. 2024EGTR: Extracting Graph from Transformer for Scene Graph Generationcited by 47
  12. 2023Panoptic Video Scene Graph Generationcited by 46
  13. 2023SceneGenie: Scene Graph Guided Diffusion Models for Image Synthesiscited by 41
  14. 2023Unbiased Scene Graph Generation in Videoscited by 39
  15. 2023Memory-and-Anticipation Transformer for Online Action Understandingcited by 39
  16. 2021Exploiting Long-Term Dependencies for Generating Dynamic Scene Graphscited by 31
  17. 2023LLM4SGG: Large Language Models for Weakly Supervised Scene Graph Generationcited by 30
  18. 2022Iterative Scene Graph Generation with Generative Transformerscited by 22
  19. 2023Accurate and Fast Compressed Video Captioningcited by 22
  20. 2023HIG: Hierarchical Interlacement Graph Approach to Scene Graph Generation in Video Understandingcited by 22
  21. 2023Learning Situation Hyper-Graphs for Video Question Answeringcited by 16
  22. 2022Hierarchical Memory Learning for Fine-Grained Scene Graph Generationcited by 15
  23. 2023MELTR: Meta Loss Transformer for Learning to Fine-tune Video Foundation Modelscited by 15
  24. 2024DSGG: Dense Relation Transformer for an End-to-end Scene Graph Generationcited by 11
  25. 2024CYCLO: Cyclic Graph Transformer Approach to Multi-Object Relationship Modeling in Aerial Videoscited by 11
  26. 2024Towards Scene Graph Anticipationcited by 10
  27. 2022Deep Hypergraph Structure Learningcited by 7
  28. 2024Multi-object event graph representation learning for Video Question Answeringcited by 2
  29. 2022Modeling Semantic Composition with Syntactic Hypergraph for Video Question Answeringcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 15.

19 papers.

Sort:

By area:

  1. 2025Directed-Tokens: A Robust Multi-Modality Alignment Approach to Large Language-Vision Modelscited by 5
  2. 2026GraphThinker: Reinforcing Temporally Grounded Video Reasoning with Event Graph Thinkingcited by 5
  3. 2025DyGEnc: Encoding a Sequence of Textual Scene Graphs to Reason and Answer Questions in Dynamic Scenescited by 4
  4. 2025MANGO: Multimodal Attention-based Normalizing Flow Approach to Fusion Learningcited by 3
  5. 2026Synthetic Visual Genome 2: Extracting Large-scale Spatio-Temporal Scene Graphs from Videoscited by 3
  6. 2026FocusGraph: Graph-Structured Frame Selection for Embodied Long Video Question Answeringcited by 3
  7. 2026φ-DPO: Fairness Direct Preference Optimization Approach to Continual Learning in Large Multimodal Modelscited by 2
  8. 2026Response-G1: Explicit Scene Graph Modeling for Proactive Streaming Video Understandingcited by 2
  9. 2025Language-Driven Object-Oriented Two-Stage Method for Scene Graph Anticipationcited by 1
  10. 2026Learning to Evolve Scenes: Reasoning about Human Activities with Scene Graphscited by 1
  11. 2026SimGraph: A Unified Framework for Scene Graph-Based Image Generation and Editingcited by 0
  12. 2026A Survey on Semantic Communication for Vision: Categories, Frameworks, Enabling Techniques, and Applicationscited by 0
7 more
  1. 2026Self-Correction Can Amplify Hallucinations: Fact-Level Repair with Graph-Based Evidence Routing in Multimodal Generationcited by 0
  2. 2026Learning to Deny: Action Denial in Multimodal Large Language Modelscited by 0
  3. 2026Beyond Pairwise Relations: Dynamic Manipulation Hypergraphs for Vision-Based Human Activity Recognitioncited by 0
  4. 2026SportsGrounder: Proposal-Aided Interleaved Grounding for Dense Sports Video Reasoningcited by 0
  5. 2026OvDSGG: End-to-End Open-Vocabulary Dynamic Scene Graph Generationcited by 0
  6. 2026SceneForesee: Learning Scene Evolution for Spatiotemporal Reasoningcited by 0
  7. 2026RTAformer: Relation-Transition-Aware Transformer for Video Scene Graph Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024SceneLLM: Implicit Language Reasoning in LLM for Dynamic Scene Graph Generation
  2. 2026HyperVis: Continuous Latent Visual Relational Graphs on the Lorentz Hyperboloid for Compositional Reasoning
  3. 2026Scene Graph Thinking: Reinforcing Structured Visual Reasoning for Multimodal Large Language Models
  4. 2024LLaVA-SpaceSGG: Visual Instruct Tuning for Open-vocabulary Scene Graph Generation with Enhanced Spatial Relations
  5. 2026HyperGVL: Benchmarking and Improving Large Vision-Language Models in Hypergraph Understanding and Reasoning
  6. 2025A Closer Look at Dynamic Scene Graph Generation In the Era of Multimodal Large Language Models
  7. 2026SceneGraphVLM: Dynamic Scene Graph Generation from Video with Vision-Language Models
  8. 20243DGraphLLM: Combining Semantic Graphs and Large Language Models for 3D Scene Understanding