mlsanityData to 14 Oct 2026

SceneGraphVLM: Dynamic Scene Graph Generation from Video with Vision-Language Models

arXiv 2605.13667 ↗First posted 13 May 2026cs.CVcited by 1

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Scene graph generation provides a compact structured representation for visual perception, but accurate and fast graph prediction from images and videos remains challenging. Recent VLM-based methods can generate scene graphs end-to-end as structured text, yet often produce long outputs with irrelevant objects and relations. We present SceneGraphVLM, a compact method for image and video scene graph generation with small visual language models. SceneGraphVLM serializes graphs in a token-efficient TOON format and trains the model in two stages: supervised fine-tuning followed by reinforcement learning with hallucination-aware rewards that balance relation coverage and precision while penalizing unsupported objects and relations. For videos, the model can optionally condition each frame on the previously generated graph, providing lightweight short-term context without tracking or post-processing. We evaluate SceneGraphVLM on PSG, PVSG, and Action Genome. With compact VLMs and vLLM-accelerated decoding, SceneGraphVLM achieves a strong quality-speed trade-off, improves precision-oriented SGG metrics while preserving reasonable recall, and generates complete scene graphs with approximately one-second latency. Code and implementation details are available at: https://github.com/markus0440/SceneGraphVLM.git.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 37 of its 44 references that are in the map. First posted on arXiv 13 May 2026.

Builds on

37 of 44 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Open World Scene Graph Generation using Vision Language Models9 Jun
  2. 2024From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Models1 Apr

Other references · 35

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  5. 2025Qwen3-VL Technical Reportcited by 2,793
  6. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  7. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  8. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  9. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  10. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
  11. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  12. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
23 more
  1. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 572
  2. 2017Scene Graph Generation by Iterative Message Passingcited by 462
  3. 2017Neural Motifs: Scene Graph Parsing with Global Contextcited by 414
  4. 2023GPTScore: Evaluate as You Desirecited by 361
  5. 2024SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuningcited by 355
  6. 2016Visual Relationship Detection with Language Priorscited by 334
  7. 2018Learning to Compose Dynamic Tree Structures for Visual Contextscited by 259
  8. 2019Action Genome: Actions as Composition of Spatio-temporal Scene Graphscited by 250
  9. 2020GPS-Net: Graph Property Sensing Network for Scene Graph Generationcited by 166
  10. 2017Scene Graph Generation from Objects, Phrases and Region Captionscited by 154
  11. 2022Panoptic Scene Graph Generationcited by 114
  12. 2025Ovis2.5 Technical Reportcited by 108
  13. 2019Graphical Contrastive Losses for Scene Graph Parsingcited by 100
  14. 2021Spatial-Temporal Transformer for Dynamic Scene Graph Generationcited by 96
  15. 2022RelTR: Relation Transformer for Scene Graph Generationcited by 96
  16. 2021Target Adaptive Context Aggregation for Video Scene Graph Generationcited by 52
  17. 2023Panoptic Video Scene Graph Generationcited by 46
  18. 2023Unbiased Scene Graph Generation in Videoscited by 39
  19. 2021Exploiting Long-Term Dependencies for Generating Dynamic Scene Graphscited by 31
  20. 2024OED: Towards One-stage End-to-End Dynamic Scene Graph Generationcited by 23
  21. 2023Cross-Modality Time-Variant Relation Learning for Generating Dynamic Scene Graphscited by 14
  22. 2024LLaVA-SpaceSGG: Visual Instruct Tuning for Open-vocabulary Scene Graph Generation with Enhanced Spatial Relationscited by 13
  23. 2026FocusGraph: Graph-Structured Frame Selection for Embodied Long Video Question Answeringcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026OptiSight: Bridging Semantic Reasoning and Geometric Control for Embodied Navigationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Closer Look at Dynamic Scene Graph Generation In the Era of Multimodal Large Language Models
  2. 2024Generate Any Scene: Scene Graph Driven Data Synthesis for Visual Generation Training
  3. 2026SGG-Rrm 3: From Next-Token Prediction to End-to-End Unbiased Scene Graph Generation
  4. 2024SceneLLM: Implicit Language Reasoning in LLM for Dynamic Scene Graph Generation
  5. 2025DIFFVSGG: Diffusion-Driven Online Video Scene Graph Generation
  6. 2026Can We Build Scene Graphs, Not Classify Them? FlowSG: Progressive Image-Conditioned Scene Graph Generation with Flow Matching
  7. 2026Towards Spatio-Temporal World Scene Graph Generation from Monocular Videos
  8. 2026Dependency-Aware Discrete Diffusion for Scene Graph Generation