Brand-as-Memory: Vision-Language Models Encode Causal, Mechanistically Localizable Credibility Priors for News Sources
On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning
Abstract (arXiv)
Vision-language models (VLMs) increasingly read news and web content as images, where the publisher's identity is visually present. We show that VLMs carry a strong source-credibility prior keyed on outlet identity, and study it along three axes. (i) Cross-model benchmark. We introduce CueTrust, a cross-model diagnostic that measures which surface source cue overrides an article's content evidence via a Source-Override Index (SOI). Across seven VLMs and five cues, the vulnerability profile is model- and scale-dependent, and the override is outlet-identity-specific and encoding-invariant, firing from the masthead name, the logo image, or the bare domain, but not from a named author, in-text authority, or page layout (clean negative controls). (ii) Mechanistic account. For the brand cue, we give a full mechanistic account: swapping only the masthead moves credibility across an approximately 11 log-odds range that tracks professional ratings (rho = 0.88 with Media Bias/Fact Check). The prior is dual-coded (name and logo), strengthens with scale, is causally formed at layers 19-21, carried by interpretable seed-stable sparse-autoencoder features, and recurs at the same relative locus in a second model family. It overrides content (about 1.8x) as a signal-magnitude effect within a shared pathway, not a privileged route. Steering the localized direction selectively reduces the override (41% reduction) and generalizes to held-out outlets, confirming the prior is causally used, not merely decodable. Deployed VLMs may thus defer to source identity over the evidence in front of them, a reliability failure we can measure across models, localize, and causally probe. We release the stimulus suite and CueTrust.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 33 of its 41 references that are in the map. First posted on arXiv 3 Jul 2026.
Builds on
33 of 41 references.
Closest ancestors · 3
Same direction, newest first
- 2026Circuit Tracing in Vision-Language Models: Understanding the Internal Mechanisms of Multimodal Thinking23 Feb
- 2025When Seeing Overrides Knowing: Disentangling Knowledge Conflicts in Vision-Language Models18 Jul
- 2025Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMs10 Jun
Other references · 30
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2023Visual Instruction Tuningcited by 7,792
- 2025Qwen2.5-VL Technical Reportcited by 5,418
- 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
- 2022Locating and Editing Factual Associations in GPTcited by 2,609
- 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
- 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
- 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
- 2023Towards Understanding Sycophancy in Language Modelscited by 988
- 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
- 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
- 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
18 more
- 2019Designing and Interpreting Probes with Control Taskscited by 608
- 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
- 2024Scaling and evaluating sparse autoencoderscited by 530
- 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 412
- 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 381
- 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 328
- 2025SmolVLM: Redefining small and efficient multimodal modelscited by 226
- 2023Interpreting CLIP's Image Representation via Text-Based Decompositioncited by 189
- 2024Transcoders Find Interpretable LLM Feature Circuitscited by 186
- 2026Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnetcited by 175
- 2025Open Problems in Mechanistic Interpretabilitycited by 173
- 2025Sparse Autoencoders Trained on the Same Data Learn Different Featurescited by 86
- 2025SAEs Are Good for Steering -- If You Select the Right Featurescited by 74
- 2023Accuracy and Political Bias of News Source Credibility Ratings by Large Language Modelscited by 25
- 2025Does higher interpretability imply better utility? A Pairwise Analysis on Sparse Autoencoderscited by 9
- 2026Interpretable Debiasing of Vision-Language Models for Social Fairnesscited by 6
- 2025Ensembling Sparse Autoencoderscited by 3
- 2026In Agents We Trust, but Who Do Agents Trust? Latent Source Preferences Steer LLM Generationscited by 3
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Images Amplify Misinformation Sharing in Vision-Language Models
- 2025Seeing Through Deception: Uncovering Misleading Creator Intent in Multimodal News with Vision-Language Models
- 2025Large Language Model-Informed Feature Discovery Improves Prediction and Interpretation of Credibility Perceptions of Visual Content
- 2025Verifying Cross-modal Entity Consistency in News using Vision-language Models
- 2025Bias in the Picture: Benchmarking VLMs with Social-Cue News Images and LLM-as-Judge Assessment
- 2026SynCred-Bench: Benchmarking Synthetic Credibility in AI-Generated Visual Misinformation
- 2025CIDER: A Causal Cure for Brand-Obsessed Text-to-Image Models
- 2025RW-Post: Auditable Evidence-Grounded Multimodal Fact-Checking in the Wild