mlsanityData to 13 Oct 2026

Gaslight, Gatekeep, V1-V3: Early Visual Cortex Alignment Shields Vision-Language Models from Sycophantic Manipulation

arXiv 2604.13803 ↗First posted 15 Apr 2026cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-language models are increasingly deployed in high-stakes settings, yet their susceptibility to sycophantic manipulation remains poorly understood, particularly in relation to how these models represent visual information internally. Whether models whose visual representations more closely mirror human neural processing are also more resistant to adversarial pressure is an open question with implications for both neuroscience and AI safety. We investigate this question by evaluating 12 open-weight vision-language models spanning 6 architecture families and a 40× parameter range (256M--10B) along two axes: brain alignment, measured by predicting fMRI responses from the Natural Scenes Dataset across 8 human subjects and 6 visual cortex regions of interest, and sycophancy, measured through 76,800 two-turn gaslighting prompts spanning 5 categories and 10 difficulty levels. Region-of-interest analysis reveals that alignment specifically in early visual cortex (V1--V3) is a reliable negative predictor of sycophancy (r = -0.441, BCa 95% CI [-0.740, -0.031]), with all 12 leave-one-out correlations negative and the strongest effect for existence denial attacks (r = -0.597, p = 0.040). This anatomically specific relationship is absent in higher-order category-selective regions, suggesting that faithful low-level visual encoding provides a measurable anchor against adversarial linguistic override in vision-language models. We release our code on hrefhttps://github.com/aryashah2k/Gaslight-Gatekeep-Sycophantic-ManipulationGitHub and dataset on hrefhttps://huggingface.co/datasets/aryashah00/Gaslight-Gatekeep-V1-V3Hugging Face

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 24 of its 47 references that are in the map. First posted on arXiv 15 Apr 2026.

Builds on

24 of 47 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024What matters when building vision-language models?3 May

Other references · 23

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,550
  3. 2022Training language models to follow instructions with human feedbackcited by 15,856
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,103
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,389
  6. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,191
  7. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,949
  8. 2020Shortcut Learning in Deep Neural Networkscited by 1,944
  9. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,709
  10. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  11. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 751
  12. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 723
11 more
  1. 2023Visual Adversarial Examples Jailbreak Aligned Large Language Modelscited by 326
  2. 2023On Evaluating Adversarial Robustness of Large Vision-Language Modelscited by 284
  3. 2023MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Modelscited by 272
  4. 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attackscited by 192
  5. 2024Images are Achilles' Heel of Alignment: Exploiting Visual Vulnerabilities for Jailbreaking Multimodal Large Language Modelscited by 133
  6. 2024Language Models Learn to Mislead Humans via RLHFcited by 119
  7. 2023Improving neural network representations using human similarity judgmentscited by 38
  8. 2023The Algonauts Project 2023 Challenge: How the Human Brain Makes Sense of Natural Scenescited by 34
  9. 2023Alignment with human representations supports robust few-shot learningcited by 25
  10. 2024Understanding the Effects of Iterative Prompting on Truthfulnesscited by 14
  11. 2025Alignment and Adversarial Robustness: Are More Human-Like Models More Secure?cited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Do VLMs Align Better with Humans than LLMs during Natural Reading?
  2. 2024Vision-Language Models Align with Human Neural Representations in Concept Processing
  3. 2026SycoPhantasy: Quantifying Sycophancy and Hallucination in Small Open Weight VLMs for Vision-Language Scoring of Fantasy Characters
  4. 2025Brain-Language Model Alignment: Insights into the Platonic Hypothesis and Intermediate-Layer Advantage
  5. 2026Do Models See in Line with Human Vision? Probing the Correspondence Between LVLM Representations and EEG Signals
  6. 2026Linguistic properties and model scale in brain encoding: from small to compressed language models
  7. 2024ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time
  8. 2026Brain alignment of reasoning and action representations from vision-language and action models during naturalistic gameplay