mlsanityData to 13 Oct 2026

Teaching Structured Vision&Language Concepts to Vision&Language Models

arXiv 2211.11733 ↗First posted 21 Nov 2022cs.CVcited by 71

Not on the map. Why?

Cited by 33 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Vision and Language (VL) models have demonstrated remarkable zero-shot performance in a variety of tasks. However, some aspects of complex language understanding still remain a challenge. We introduce the collective notion of Structured Vision&Language Concepts (SVLC) which includes object attributes, relations, and states which are present in the text and visible in the image. Recent studies have shown that even the best VL models struggle with SVLC. A possible way of fixing this issue is by collecting dedicated datasets for teaching each SVLC type, yet this might be expensive and time-consuming. Instead, we propose a more elegant data-driven approach for enhancing VL models' understanding of SVLCs that makes more effective use of existing VL pre-training datasets and does not require any additional data. While automatic understanding of image structure still remains largely unsolved, language structure is much better modeled and understood, allowing for its effective utilization in teaching VL models. In this paper, we propose various techniques based on language structure understanding that can be used to manipulate the textual part of off-the-shelf paired VL datasets. VL models trained with the updated data exhibit a significant improvement of up to 15% in their SVLC understanding with only a mild degradation in their zero-shot capabilities both when training from scratch or fine-tuning a pre-trained model.

Key facts. Cited by 71 papers in the map (data through 13 Oct 2026). Builds on 70 of its 87 references that are in the map. First posted on arXiv 21 Nov 2022.

Builds on

70 of 87 references.

References · 70

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,550
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,231
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,322
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,009
  7. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,476
  8. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,170
  9. 2016Overcoming catastrophic forgetting in neural networkscited by 4,753
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,314
  11. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,281
  12. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,855
38 more
  1. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,882
  2. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,649
  3. 2016Learning without Forgettingcited by 2,479
  4. 2015Character-level Convolutional Networks for Text Classificationcited by 2,319
  5. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,266
  6. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,082
  7. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,616
  8. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,551
  9. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,548
  10. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,492
  11. 2018Relational inductive biases, deep learning, and graph networkscited by 1,353
  12. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,294
  13. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,227
  14. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  15. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,058
  16. 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 802
  17. 2018AutoAugment: Learning Augmentation Policies from Datacited by 730
  18. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 706
  19. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 627
  20. 2018End-to-End Incremental Learningcited by 597
  21. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 525
  22. 2017Scene Graph Generation by Iterative Message Passingcited by 462
  23. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
  24. 2017Measuring Catastrophic Forgetting in Neural Networkscited by 360
  25. 2018Image Generation from Scene Graphscited by 324
  26. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 320
  27. 2022Generating Sequences by Learning to Self-Correctcited by 273
  28. 2019Action Genome: Actions as Composition of Spatio-temporal Scene Graphscited by 249
  29. 2018Videos as Space-Time Region Graphscited by 221
  30. 2022Vision-Language Pre-Training with Triple Contrastive Learningcited by 212
  31. 2022WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creationcited by 201
  32. 2020Data Augmentation using Pre-trained Transformer Modelscited by 168
  33. 2020Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semanticscited by 159
  34. 2022ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Modelscited by 141
  35. 2022CyCLIP: Cyclic Contrastive Language-Image Pretrainingcited by 127
  36. 2020DRG: Dual Relation Graph for Human-Object Interaction Detectioncited by 117
  37. 2022VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relationscited by 114
  38. 2019Something-Else: Compositional Action Recognition with Spatial-Temporal Interaction Networkscited by 112

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 23; 2024: 23; 2025: 19; 2026: 6.

71 papers.

Sort:

By area:

  1. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 652
  2. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionalitycited by 239
  3. 2023TMR: Text-to-Motion Retrieval Using Contrastive 3D Human Motion Synthesiscited by 198
  4. 2023Compositional Chain-of-Thought Prompting for Large Multimodal Modelscited by 163
  5. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 81
  6. 2024Open3DSG: Open-Vocabulary 3D Scene Graphs from Point Clouds with Queryable Objects and Open-Set Relationshipscited by 76
  7. 2023Dense and Aligned Captions (DAC) Promote Compositional Reasoning in VL Modelscited by 72
  8. 2025Vision-Language-Action (VLA) Models: Concepts, Progress, Applications and Challengescited by 72
  9. 2024TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negativescited by 57
  10. 2023Revisiting the Role of Language Priors in Vision-Language Modelscited by 50
  11. 2023Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understandingcited by 46
  12. 2023Paxion: Patching Action Knowledge in Video-Language Foundation Modelscited by 42
38 more
  1. 2024SUGARCREPE++ Dataset: Vision-Language Model Sensitivity to Semantic and Lexical Alterationscited by 42
  2. 2023Incorporating Structured Representations into Pretrained Vision & Language Models Using Scene Graphscited by 39
  3. 2023Going Beyond Nouns With Vision & Language Models Using Synthetic Datacited by 33
  4. 2023Coarse-to-Fine Contrastive Learning in Image-Text-Graph Space for Improved Vision-Language Compositionalitycited by 32
  5. 2023DesCo: Learning Object Recognition with Rich Language Descriptionscited by 25
  6. 2024Towards Multimodal In-Context Learning for Vision & Language Modelscited by 24
  7. 2024VisMin: Visual Minimal-Change Understandingcited by 23
  8. 2023Zero-shot Referring Expression Comprehension via Structural Similarity Between Images and Captionscited by 21
  9. 2023CoVLM: Composing Visual Entities and Relationships in Large Language Models Via Communicative Decodingcited by 20
  10. 2023Lang3DSG: Language-based contrastive pre-training for 3D Scene Graph predictioncited by 18
  11. 2025Post-pre-training for Modality Alignment in Vision-Language Foundation Modelscited by 18
  12. 2023Probing Conceptual Understanding of Large Visual-Language Modelscited by 17
  13. 2024VladVA: Discriminative Fine-tuning of LVLMscited by 15
  14. 2023ProTeCt: Prompt Tuning for Taxonomic Open Set Classificationcited by 12
  15. 2025Enhancing Vision-Language Compositional Understanding with Multimodal Synthetic Datacited by 12
  16. 2023Generating Enhanced Negatives for Training Language-Based Object Detectorscited by 11
  17. 2025VideoComp: Advancing Fine-Grained Compositional and Temporal Alignment in Video-Text Modelscited by 11
  18. 2025Taxonomy-Aware Evaluation of Vision-Language Modelscited by 11
  19. 2024MMRel: Benchmarking Relation Understanding in Multi-Modal Large Language Modelscited by 10
  20. 2025SCA3D: Enhancing Cross-modal 3D Retrieval via 3D Shape and Caption Paired Data Augmentationcited by 9
  21. 2025TTRV: Test-Time Reinforcement Learning for Vision Language Modelscited by 9
  22. 2024Removing Distributional Discrepancies in Captions Improves Image-Text Alignmentcited by 8
  23. 20233VL: Using Trees to Improve Vision-Language Models' Interpretabilitycited by 7
  24. 2024Diagnosing the Compositional Knowledge of Vision Language Models from a Game-Theoretic Viewcited by 7
  25. 2025VSC: Visual Search Compositional Text-to-Image Diffusion Modelcited by 7
  26. 2023ClipSitu: Effectively Leveraging CLIP for Conditional Predictions in Situation Recognitioncited by 6
  27. 2024Enhancing Conceptual Understanding in Multimodal Contrastive Learning through Hard Negative Samplescited by 6
  28. 2024Finetuning CLIP to Reason about Pairwise Differencescited by 6
  29. 2024GLOV: Guided Large Language Models as Implicit Optimizers for Vision Language Modelscited by 6
  30. 2024Teaching VLMs to Localize Specific Objects from In-context Examplescited by 6
  31. 2024Causal Graphical Models for Vision-Language Compositional Understandingcited by 6
  32. 2025What Changed and What Could Have Changed? State-Change Counterfactuals for Procedure-Aware Video Representation Learningcited by 6
  33. 2023Large Language Models Meet Computer Vision: A Brief Surveycited by 5
  34. 2025Advancing Compositional Awareness in CLIP with Efficient Fine-Tuningcited by 5
  35. 2025VITRIX-CLIPIN: Enhancing Fine-Grained Visual Understanding in CLIP via Instruction Editing Data and Long Captionscited by 5
  36. 2025Common Data Properties Limit Object-Attribute Binding in CLIPcited by 4
  37. 2025VisualOverload: Probing Visual Understanding of VLMs in Really Dense Scenescited by 4
  38. 2024Weak-to-Strong Compositional Learning from Generative Models for Language-based Object Detectioncited by 3

50 most cited of 71 All 71 (JSON)