mlsanityData to 7 Oct 2026

Can Linguistic Knowledge Improve Multimodal Alignment in Vision-Language Pretraining?

arXiv 2308.12898 ↗First posted 24 Aug 2023cs.MM cs.AI cs.CLcited by 21

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

The multimedia community has shown a significant interest in perceiving and representing the physical world with multimodal pretrained neural network models, and among them, the visual-language pertaining (VLP) is, currently, the most captivating topic. However, there have been few endeavors dedicated to the exploration of 1) whether essential linguistic knowledge (e.g., semantics and syntax) can be extracted during VLP, and 2) how such linguistic knowledge impact or enhance the multimodal alignment. In response, here we aim to elucidate the impact of comprehensive linguistic knowledge, including semantic expression and syntactic structure, on multimodal alignment. Specifically, we design and release the SNARE, the first large-scale multimodal alignment probing benchmark, to detect the vital linguistic components, e.g., lexical, semantic, and syntax knowledge, containing four tasks: Semantic structure, Negation logic, Attribute ownership, and Relationship composition. Based on our proposed probing benchmarks, our holistic analyses of five advanced VLP models illustrate that the VLP model: i) shows insensitivity towards complex syntax structures and relies on content words for sentence comprehension; ii) demonstrates limited comprehension of combinations between sentences and negations; iii) faces challenges in determining the presence of actions or spatial relationships within visual information and struggles with verifying the correctness of triple combinations. We make our benchmark and code available at urlhttps://github.com/WangFei-2019/SNARE/.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 66 of its 81 references that are in the map. First posted on arXiv 24 Aug 2023.

Builds on

66 of 81 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  2. 2023A Survey on Multimodal Large Language Models23 Jun
  3. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun

Other references · 63

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  9. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  10. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  11. 2023A Survey of Large Language Modelscited by 2,537
  12. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
38 more
  1. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  2. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  3. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  4. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  5. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  6. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  7. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 553
  8. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  9. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 452
  10. 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 317
  11. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 255
  12. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionalitycited by 238
  13. 2021Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Littlecited by 176
  14. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  15. 2023Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERTcited by 155
  16. 2023Towards Making the Most of ChatGPT for Machine Translationcited by 138
  17. 2022VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relationscited by 113
  18. 2021Probing Image-Language Transformers for Verb Understandingcited by 109
  19. 2017FOIL it! Find One mismatch between Image and Language captioncited by 95
  20. 2021Learning to Predict Visual Attributes in the Wildcited by 95
  21. 2020Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Modelscited by 90
  22. 2023Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Modelscited by 87
  23. 2020Grounded Situation Recognitioncited by 77
  24. 2020Out of Order: How Important Is The Sequential Order of Words in a Sentence in Natural Language Understanding Tasks?cited by 72
  25. 2020Multimodal Pretraining Unmasked: A Meta-Analysis and a Unified Framework of Vision-and-Language BERTscited by 71
  26. 2021Vision-and-Language or Vision-for-Language? On Cross-Modal Influence in Multimodal Transformerscited by 71
  27. 2020UnNatural Language Inferencecited by 68
  28. 2020Understanding and Improving Lexical Choice in Non-Autoregressive Translationcited by 55
  29. 2019Probing the Need for Visual Context in Multimodal Machine Translationcited by 54
  30. 2021What Context Features Can Transformer Language Models Use?cited by 49
  31. 2022Parameter-Efficient and Student-Friendly Knowledge Distillationcited by 47
  32. 2022Things not Written in Text: Exploring Spatial Commonsense from Visual Signalscited by 41
  33. 2020Seeing past words: Testing the cross-modal capabilities of pretrained V&L models on counting taskscited by 34
  34. 2019HAKE: Human Activity Knowledge Enginecited by 32
  35. 2022On Vision Features in Multimodal Machine Translationcited by 31
  36. 2022HAKE: A Knowledge Engine Foundation for Human Activity Understandingcited by 28
  37. 2020Self-Attention with Cross-Lingual Position Representationcited by 27
  38. 2020Context-Aware Cross-Attention for Non-Autoregressive Translationcited by 26

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 11; 2025: 5; 2026: 4.

21 papers.

Sort:

By area:

  1. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decodingcited by 198
  2. 2024SUGARCREPE++ Dataset: Vision-Language Model Sensitivity to Semantic and Lexical Alterationscited by 42
  3. 2025Retrieval-Augmented Perception: High-Resolution Image Perception Meets Visual RAGcited by 22
  4. 2024HEMM: Holistic Evaluation of Multimodal Foundation Modelscited by 21
  5. 2024WisdoM: Improving Multimodal Sentiment Analysis by Fusing Contextual World Knowledgecited by 16
  6. 2024Can 3D Vision-Language Models Truly Understand Natural Language?cited by 12
  7. 2025Enhancing Vision-Language Compositional Understanding with Multimodal Synthetic Datacited by 12
  8. 2023Visual Data-Type Understanding does not emerge from Scaling Vision-Language Modelscited by 9
  9. 2024Unified Lexical Representation for Interpretable Visual-Language Alignmentcited by 8
  10. 2024Simultaneous Computation and Memory Efficient Zeroth-Order Optimizer for Fine-Tuning Large Language Modelscited by 8
  11. 2025Layer as Puzzle Pieces: Compressing Large Language Models through Layer Concatenationcited by 6
  12. 2026The Paradigm Shift: A Comprehensive Survey on Large Vision Language Models for Multimodal Fake News Detectioncited by 6
9 more
  1. 2024Anchors Aweigh! Sail for Optimal Unified Multi-Modal Representationscited by 4
  2. 2024Distilled Transformers with Locally Enhanced Global Representations for Face Forgery Detectioncited by 4
  3. 2024Seeing Syntax: Uncovering Syntactic Learning Limitations in Vision-Language Modelscited by 3
  4. 2024VISLA Benchmark: Evaluating Embedding Sensitivity to Semantic and Lexical Alterationscited by 1
  5. 2025Leveraging NTPs for Efficient Hallucination Detection in VLMscited by 1
  6. 2025Large model retrieval enhancement framework for construction site risk identificationcited by 0
  7. 2026DO-Bench: An Attributable Benchmark for Diagnosing Object Hallucination in Vision-Language Modelscited by 0
  8. 2026Towards High-Resolution Visual Perception via Hierarchical Entity Explorationcited by 0
  9. 2026Does Linguistic Structure Enrichment Enhance Coherence Assessment? Not With Current Architecturescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models
  2. 2026The Alignment Illusion in Multimodal Large Language Models
  3. 2025Perceiving Beyond Language Priors: Enhancing Visual Comprehension and Attention in Multimodal Models
  4. 2025Visual Representation Alignment for Multimodal Large Language Models
  5. 2024Implicit Multimodal Alignment: On the Generalization of Frozen LLMs to Multimodal Inputs
  6. 2025Decoupled Proxy Alignment: Mitigating Language Prior Conflict for Multimodal Alignment in MLLM
  7. 2025BASIC: Boosting Visual Alignment with Intrinsic Refined Embeddings in Multimodal Large Language Models
  8. 2022Do Vision-and-Language Transformers Learn Grounded Predicate-Noun Dependencies?