mlsanityData to 7 Oct 2026

Brand-as-Memory: Vision-Language Models Encode Causal, Mechanistically Localizable Credibility Priors for News Sources

arXiv 2607.03365 ↗First posted 3 Jul 2026cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-language models (VLMs) increasingly read news and web content as images, where the publisher's identity is visually present. We show that VLMs carry a strong source-credibility prior keyed on outlet identity, and study it along three axes. (i) Cross-model benchmark. We introduce CueTrust, a cross-model diagnostic that measures which surface source cue overrides an article's content evidence via a Source-Override Index (SOI). Across seven VLMs and five cues, the vulnerability profile is model- and scale-dependent, and the override is outlet-identity-specific and encoding-invariant, firing from the masthead name, the logo image, or the bare domain, but not from a named author, in-text authority, or page layout (clean negative controls). (ii) Mechanistic account. For the brand cue, we give a full mechanistic account: swapping only the masthead moves credibility across an approximately 11 log-odds range that tracks professional ratings (rho = 0.88 with Media Bias/Fact Check). The prior is dual-coded (name and logo), strengthens with scale, is causally formed at layers 19-21, carried by interpretable seed-stable sparse-autoencoder features, and recurs at the same relative locus in a second model family. It overrides content (about 1.8x) as a signal-magnitude effect within a shared pathway, not a privileged route. Steering the localized direction selectively reduces the override (41% reduction) and generalizes to held-out outlets, confirming the prior is causally used, not merely decodable. Deployed VLMs may thus defer to source identity over the evidence in front of them, a reliability failure we can measure across models, localize, and causally probe. We release the stimulus suite and CueTrust.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 33 of its 41 references that are in the map. First posted on arXiv 3 Jul 2026.

Builds on

33 of 41 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026Circuit Tracing in Vision-Language Models: Understanding the Internal Mechanisms of Multimodal Thinking23 Feb
  2. 2025When Seeing Overrides Knowing: Disentangling Knowledge Conflicts in Vision-Language Models18 Jul
  3. 2025Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMs10 Jun

Other references · 30

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2025Qwen2.5-VL Technical Reportcited by 5,418
  4. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  5. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  6. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
  7. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  8. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  9. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  10. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  11. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  12. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
18 more
  1. 2019Designing and Interpreting Probes with Control Taskscited by 608
  2. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  3. 2024Scaling and evaluating sparse autoencoderscited by 530
  4. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 412
  5. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 381
  6. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 328
  7. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 226
  8. 2023Interpreting CLIP's Image Representation via Text-Based Decompositioncited by 189
  9. 2024Transcoders Find Interpretable LLM Feature Circuitscited by 186
  10. 2026Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnetcited by 175
  11. 2025Open Problems in Mechanistic Interpretabilitycited by 173
  12. 2025Sparse Autoencoders Trained on the Same Data Learn Different Featurescited by 86
  13. 2025SAEs Are Good for Steering -- If You Select the Right Featurescited by 74
  14. 2023Accuracy and Political Bias of News Source Credibility Ratings by Large Language Modelscited by 25
  15. 2025Does higher interpretability imply better utility? A Pairwise Analysis on Sparse Autoencoderscited by 9
  16. 2026Interpretable Debiasing of Vision-Language Models for Social Fairnesscited by 6
  17. 2025Ensembling Sparse Autoencoderscited by 3
  18. 2026In Agents We Trust, but Who Do Agents Trust? Latent Source Preferences Steer LLM Generationscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Images Amplify Misinformation Sharing in Vision-Language Models
  2. 2025Seeing Through Deception: Uncovering Misleading Creator Intent in Multimodal News with Vision-Language Models
  3. 2025Large Language Model-Informed Feature Discovery Improves Prediction and Interpretation of Credibility Perceptions of Visual Content
  4. 2025Verifying Cross-modal Entity Consistency in News using Vision-language Models
  5. 2025Bias in the Picture: Benchmarking VLMs with Social-Cue News Images and LLM-as-Judge Assessment
  6. 2026SynCred-Bench: Benchmarking Synthetic Credibility in AI-Generated Visual Misinformation
  7. 2025CIDER: A Causal Cure for Brand-Obsessed Text-to-Image Models
  8. 2025RW-Post: Auditable Evidence-Grounded Multimodal Fact-Checking in the Wild