mlsanityData to 7 Oct 2026

SLIP: Structural-aware Language-Image Pretraining for Vision-Language Alignment

arXiv 2511.03019 ↗First posted 4 Nov 2025cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-Language Pretraining (VLP) has achieved remarkable success across various downstream tasks, but such gains are largely driven by scaling up on training data. Yet, literature methods treat image-text pairs as isolated training examples; this neglects the rich relational structure naturally present in many domains, such as e-commerce product co-purchase graphs and social recommendation networks. Inspired by neuroscientific evidence that human encodes knowledge as relationship cognitive maps, we introduce Structure-aware Language-Image Pretraining (SLIP). SLIP integrates a structural contrastive loss to align modalities while also modeling relationships between neighboring entities in a structured graph. To support this paradigm, we construct a large-scale Amazon Product Co-purchase Multimodal Graph Dataset, enabling structured cross-modality supervision at scale. Experiment results show that SLIP consistently outperforms CLIP on cross-modal retrieval and classification tasks in both zero-shot and few-shot settings, showing the value of relational supervision for cross-modal alignment.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 23 of its 27 references that are in the map. First posted on arXiv 4 Nov 2025.

Builds on

23 of 27 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Structure-CLIP: Towards Scene Graph Knowledge to Enhance Multi-modal Structured Representations6 May
  2. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trends17 Oct
  3. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation28 Jan
  4. 2021FILIP: Fine-grained Interactive Language-Image Pre-Training9 Nov
  5. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervision24 Aug

Other references · 18

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2017Graph Attention Networkscited by 8,854
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  5. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  6. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  7. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  8. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  9. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  10. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  11. 2018Exploring Visual Relationship for Image Captioningcited by 215
  12. 2024Bridging Language and Items for Retrieval and Recommendation: Benchmarking LLMs as Semantic Encoderscited by 213
6 more
  1. 2020ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graphcited by 211
  2. 2019Visual Semantic Reasoning for Image-Text Matchingcited by 185
  3. 2020Graph-Structured Referring Expression Reasoning in The Wildcited by 58
  4. 2022Uni-Perceiver v2: A Generalist Model for Large-Scale Vision and Vision-Language Taskscited by 57
  5. 2022VQA-GNN: Reasoning with Multimodal Knowledge via Graph Neural Networks for Visual Question Answeringcited by 22
  6. 2024Learning on Multimodal Graphs: A Surveycited by 22

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Can Linguistic Knowledge Improve Multimodal Alignment in Vision-Language Pretraining?
  2. 2025Seeing Through Words, Speaking Through Pixels: Deep Representational Alignment Between Vision and Language Models
  3. 2026VL-KGE: Vision-Language Models Meet Knowledge Graph Embeddings
  4. 2022Single-Stream Multi-Level Alignment for Vision-Language Pretraining
  5. 2026Supervised Classification Heads as Semantic Prototypes: Unlocking Vision-Language Alignment via Weight Recycling
  6. 2022Cross-modal Attention Congruence Regularization for Vision-Language Relation Alignment
  7. 2021Probing Inter-modality: Visual Parsing with Self-Attention for Vision-Language Pre-training
  8. 2025TinyAlign: Boosting Lightweight Vision-Language Models by Mitigating Modal Alignment Bottlenecks