mlsanityData to 7 Oct 2026

Incorporating Structured Representations into Pretrained Vision & Language Models Using Scene Graphs

arXiv 2305.06343 ↗First posted 10 May 2023cs.CVcited by 39

Not on the map. Why?

Cited by 24 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Vision and language models (VLMs) have demonstrated remarkable zero-shot (ZS) performance in a variety of tasks. However, recent works have shown that even the best VLMs struggle to capture aspects of compositional scene understanding, such as object attributes, relations, and action states. In contrast, obtaining structured annotations, such as scene graphs (SGs), that could improve these models is time-consuming and costly, and thus cannot be used on a large scale. Here we ask whether small SG datasets can provide sufficient information for enhancing structured understanding of pretrained VLMs. We show that it is indeed possible to improve VLMs when learning from SGs by integrating components that incorporate structured information into both visual and textual representations. For the visual side, we incorporate a special "SG Component" in the image transformer trained to predict SG information, while for the textual side, we utilize SGs to generate fine-grained captions that highlight different compositional aspects of the scene. Our method improves the performance of several popular VLMs on multiple VL datasets with only a mild degradation in ZS capabilities.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Builds on 85 of its 99 references that are in the map. First posted on arXiv 10 May 2023.

Builds on

85 of 99 references.

References · 85

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  8. 2020End-to-End Object Detection with Transformerscited by 7,384
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  10. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  12. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
38 more
  1. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  2. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  3. 2022Visual Prompt Tuningcited by 1,674
  4. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  5. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  6. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  7. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  8. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  9. 2018Relational inductive biases, deep learning, and graph networkscited by 1,341
  10. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  11. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  12. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  13. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  14. 2021Learning to Prompt for Continual Learningcited by 938
  15. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  16. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  17. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  18. 2017Learning multiple visual domains with residual adapterscited by 553
  19. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  20. 2017Scene Graph Generation by Iterative Message Passingcited by 462
  21. 2017Neural Motifs: Scene Graph Parsing with Global Contextcited by 413
  22. 2018Image Generation from Scene Graphscited by 324
  23. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 319
  24. 2018Video Action Transformer Networkcited by 316
  25. 2021Prompting Visual-Language Models for Efficient Video Understandingcited by 309
  26. 2022Visual Spatial Reasoningcited by 288
  27. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 278
  28. 2019Action Genome: Actions as Composition of Spatio-temporal Scene Graphscited by 249
  29. 2019Simple, Scalable Adaptation for Neural Machine Translationcited by 239
  30. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  31. 2018Videos as Space-Time Region Graphscited by 221
  32. 2021Training Neural Networks with Fixed Sparse Maskscited by 212
  33. 2020ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graphcited by 211
  34. 2023Teaching CLIP to Count to Tencited by 194
  35. 2022CREPE: Can Vision-Language Foundation Models Reason Compositionally?cited by 177
  36. 2019What Can Neural Networks Reason About?cited by 155
  37. 2021Towards Long-Form Video Understandingcited by 146
  38. 2022ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Modelscited by 141

50 most cited of 85 All 85 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 18; 2025: 10; 2026: 4.

39 papers.

Sort:

By area:

  1. 2023Compositional Chain-of-Thought Prompting for Large Multimodal Modelscited by 160
  2. 2023Dense and Aligned Captions (DAC) Promote Compositional Reasoning in VL Modelscited by 72
  3. 2023Revisiting the Role of Language Priors in Vision-Language Modelscited by 50
  4. 2024Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learningcited by 49
  5. 2024SUGARCREPE++ Dataset: Vision-Language Model Sensitivity to Semantic and Lexical Alterationscited by 42
  6. 2023Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understandingcited by 40
  7. 2024TraveLER: A Modular Multi-LMM Agent Framework for Video Question-Answeringcited by 24
  8. 2024VisMin: Visual Minimal-Change Understandingcited by 23
  9. 2023Zero-shot Referring Expression Comprehension via Structural Similarity Between Images and Captionscited by 21
  10. 2023Towards Grounded Visual Spatial Reasoning in Multi-Modal Vision Language Modelscited by 18
  11. 2024Preserving Multi-Modal Capabilities of Pre-trained VLMs for Improving Vision-Linguistic Compositionalitycited by 17
  12. 2024VladVA: Discriminative Fine-tuning of LVLMscited by 15
27 more
  1. 2025Enhancing Vision-Language Compositional Understanding with Multimodal Synthetic Datacited by 12
  2. 2024Enhancing Few-Shot Vision-Language Classification with Large Multimodal Model Featurescited by 11
  3. 2024Removing Distributional Discrepancies in Captions Improves Image-Text Alignmentcited by 8
  4. 2024Fact :Teaching MLLMs with Faithful, Concise and Transferable Rationalescited by 7
  5. 2024Diagnosing the Compositional Knowledge of Vision Language Models from a Game-Theoretic Viewcited by 7
  6. 2024Graph-Based Captioning: Enhancing Visual Descriptions by Interconnecting Region Captionscited by 7
  7. 2025VSC: Visual Search Compositional Text-to-Image Diffusion Modelcited by 7
  8. 2024Scene-Graph ViT: End-to-End Open-Vocabulary Visual Relationship Detectioncited by 6
  9. 2024Relation Rectification in Diffusion Modelcited by 6
  10. 2024LLaVA-SG: Leveraging Scene Graphs as Visual Semantic Expression in Vision-Language Modelscited by 6
  11. 2024Causal Graphical Models for Vision-Language Compositional Understandingcited by 6
  12. 2024Beyond Embeddings: The Promise of Visual Table in Visual Reasoningcited by 5
  13. 2025Can Video LLMs Refuse to Answer? Alignment for Answerability in Video Large Language Modelscited by 5
  14. 2023Generalized Unbiased Scene Graph Generationcited by 3
  15. 2024Path-LLM: A Shortest-Path-based LLM Learning for Unified Graph Representationcited by 2
  16. 2025SCRAMBLe : Enhancing Multimodal LLM Compositionality with Synthetic Preference Datacited by 2
  17. 2025Enhancing Compositional Reasoning in CLIP via Reconstruction and Alignment of Text Descriptionscited by 2
  18. 2024VISLA Benchmark: Evaluating Embedding Sensitivity to Semantic and Lexical Alterationscited by 1
  19. 2025Vision-Free Retrieval: Rethinking Multimodal Search with Textual Scene Descriptionscited by 1
  20. 2025COCO-Tree: Compositional Hierarchical Concept Trees for Enhanced Reasoning in Vision Language Modelscited by 1
  21. 2025Concept-RuleNet: Grounded Multi-Agent Neurosymbolic Reasoning in Vision Language Modelscited by 1
  22. 2026HyperVis: Continuous Latent Visual Relational Graphs on the Lorentz Hyperboloid for Compositional Reasoningcited by 1
  23. 2026The Gate Always Closes: On Injecting Auxiliary Signals into Frozen Vision-Language Modelscited by 1
  24. 2025REVEAL: Relation-based Video Representation Learning for Video-Question-Answeringcited by 0
  25. 2025SCLARO: A Dataset for Grounded Scenario-Level Scene Understanding and ScenarioCLIP for Benchmarkingcited by 0
  26. 2026Inference-Time Structural Reasoning for Compositional Vision-Language Understandingcited by 0
  27. 2026SynMulti: Synthetic-to-Real Learning for Multimodal Video Understandingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Structure-CLIP: Towards Scene Graph Knowledge to Enhance Multi-modal Structured Representations
  2. 2024Predicate Debiasing in Vision-Language Models Integration for Scene Graph Generation Enhancement
  3. 2025Measuring Image-Relation Alignment: Reference-Free Evaluation of VLMs and Synthetic Pre-training for Open-Vocabulary Scene Graph Generation
  4. 2025Open World Scene Graph Generation using Vision Language Models
  5. 2025PRISM-0: A Predicate-Rich Scene Graph Generation Framework for Zero-Shot Open-Vocabulary Tasks
  6. 20243D Scene Graph Guided Vision-Language Pre-training
  7. 2024Grounding Descriptions in Images informs Zero-Shot Visual Recognition
  8. 2025Interpretable Zero-Shot Learning with Locally-Aligned Vision-Language Model