mlsanityData to 14 Oct 2026

LLaVA-SG: Leveraging Scene Graphs as Visual Semantic Expression in Vision-Language Models

arXiv 2408.16224 ↗First posted 29 Aug 2024cs.CV cs.AIcited by 6

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Recent advances in large vision-language models (VLMs) typically employ vision encoders based on the Vision Transformer (ViT) architecture. The division of the images into patches by ViT results in a fragmented perception, thereby hindering the visual understanding capabilities of VLMs. In this paper, we propose an innovative enhancement to address this limitation by introducing a Scene Graph Expression (SGE) module in VLMs. This module extracts and structurally expresses the complex semantic information within images, thereby improving the foundational perception and understanding abilities of VLMs. Extensive experiments demonstrate that integrating our SGE module significantly enhances the VLM's performance in vision-language tasks, indicating its effectiveness in preserving intricate semantic details and facilitating better visual understanding.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 25 of its 29 references that are in the map. First posted on arXiv 29 Aug 2024.

Builds on

25 of 29 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Lyrics: Boosting Fine-grained Language-Vision Alignment and Comprehension via Semantic-aware Visual Objects8 Dec
  2. 2023Compositional Chain-of-Thought Prompting for Large Multimodal Models27 Nov
  3. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond24 Aug
  4. 2023Kosmos-2: Grounding Multimodal Large Language Models to the World26 Jun
  5. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan

Other references · 20

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2023Segment Anythingcited by 8,234
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  9. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  10. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  11. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  12. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
8 more
  1. 2022DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detectioncited by 1,319
  2. 2023Segment Everything Everywhere All at Oncecited by 548
  3. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 547
  4. 2023Recognize Anything: A Strong Image Tagging Modelcited by 309
  5. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 278
  6. 2023Incorporating Structured Representations into Pretrained Vision & Language Models Using Scene Graphscited by 39
  7. 2024Incorporating Visual Experts to Resolve the Information Loss in Multimodal Large Language Modelscited by 12
  8. 2023Improving Scene Graph Generation with Superpixel-Based Interaction Learningcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 2.

6 papers.

  1. 2025MMGraphRAG: Bridging Vision and Language with Interpretable Multimodal Knowledge Graphscited by 16
  2. 2024ROOT: VLM based System for Indoor Scene Understanding and Beyondcited by 6
  3. 2025PoSh: Using Scene Graphs To Guide LLMs-as-a-Judge For Detailed Image Descriptionscited by 1
  4. 2025ZING-3D: Zero-shot Incremental 3D Scene Graphs via Vision-Language Modelscited by 1
  5. 2026HyperVis: Continuous Latent Visual Relational Graphs on the Lorentz Hyperboloid for Compositional Reasoningcited by 1
  6. 2026Frequency-guided Multi-level Reasoning for Scene Graph Generation in Videocited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Rethinking Visual Information Processing in Multimodal LLMs
  2. 2024From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Models
  3. 2023Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding
  4. 2024Benchmarking and Improving Large Vision-Language Models for Fundamental Visual Graph Understanding and Reasoning
  5. 2025Hidden in plain sight: VLMs overlook their visual representations
  6. 2024How Well Can Vision Language Models See Image Details?
  7. 2025Weaving Context Across Images: Improving Vision-Language Models through Focus-Centric Visual Chains
  8. 2025LLaVA3: Representing 3D Scenes like a Cubist Painter to Boost 3D Scene Understanding of VLMs