mlsanityData to 7 Oct 2026

Towards Human Cognition: Visual Context Guides Syntactic Priming in Fusion-Encoded Models

arXiv 2502.17669 ↗First posted 24 Feb 2025cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Structural priming is a cognitive phenomenon where exposure to a particular syntactic structure increases the likelihood of producing the same structure in subsequent utterances. While humans consistently demonstrate structural priming effects across various linguistic contexts, it remains unclear whether multimodal large language models (MLLMs) exhibit similar syntactic preservation behaviors. We introduce PRISMATIC, the first multimodal structural priming dataset, which advances computational linguistics by providing a standardized benchmark for investigating syntax-vision interactions. We propose the Syntactic Preservation Index (SPI), a novel reference-free evaluation metric designed specifically to assess structural priming effects in sentence level. Using this metric, we constructed and tested models with two different multimodal encoding architectures to investigate their structural preservation capabilities. Our experimental results demonstrate that models with both encoding methods show comparable syntactic priming effects. However, only fusion-encoded models exhibit robust positive correlations between priming effects and visual similarity, suggesting a cognitive process more aligned with human psycholinguistic patterns. This work provides new insights into evaluating and understanding how syntactic information is processed in multimodal language models.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 19 of its 43 references that are in the map. First posted on arXiv 24 Feb 2025.

Builds on

19 of 43 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training14 Mar

Other references · 18

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  6. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  7. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  8. 2015Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networkscited by 305
  9. 2020Syntactic Structure from Deep Learningcited by 99
  10. 2023CoEdIT: Text Editing by Task-Specific Instruction Tuningcited by 67
  11. 2021Structural Persistence in Language Models: Priming as a Window into Abstract Language Representationscited by 44
  12. 2023Structure-CLIP: Towards Scene Graph Knowledge to Enhance Multi-modal Structured Representationscited by 37
6 more
  1. 2019Using Priming to Uncover the Organization of Syntactic Representations in Neural Language Modelscited by 29
  2. 2024Do Language Models Exhibit Human-like Structural Priming Effects?cited by 21
  3. 2021Probing Multimodal Embeddings for Linguistic Properties: the Visual-Semantic Casecited by 20
  4. 2023Structural Priming Demonstrates Abstract Grammatical Representations in Multilingual Language Modelscited by 16
  5. 2022Do Vision-and-Language Transformers Learn Grounded Predicate-Noun Dependencies?cited by 8
  6. 2024Modeling Bilingual Sentence Processing: Evaluating RNN and Transformer Architectures for Cross-Language Structural Primingcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Patterns of Priming in Production: Lexical, Semantic and Structural Alignment in Language Model Generation
  2. 2026STRUCTURALCOST: A controlled reading time dataset for modeling human sentence processing difficulty
  3. 2026Architecture-Dependent Fusion Pathways in MLLMs
  4. 2024Seeing Syntax: Uncovering Syntactic Learning Limitations in Vision-Language Models
  5. 2026Dual Alignment Between Language Model Layers and Human Sentence Processing
  6. 2025Understanding Language Prior of LVLMs by Contrasting Chain-of-Embedding
  7. 2024DevBench: A multimodal developmental benchmark for language learning
  8. 2026Where Should Language Sit in a Multimodal Model? Lessons from What Language Does to Human Perception and Cognition