mlsanityData to 7 Oct 2026

Finding Culture-Sensitive Neurons in Vision-Language Models

arXiv 2510.24942 ↗First posted 28 Oct 2025cs.LG cs.AI cs.CLcited by 3

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Despite their impressive performance, vision-language models (VLMs) still struggle on culturally situated inputs. To understand how VLMs process culturally grounded information, we study the presence of culture-sensitive neurons, i.e., neurons whose activations show preferential sensitivity to inputs associated with particular cultural contexts. We examine whether such neurons are important for culturally diverse visual question answering and where they are located. Using the CVQA benchmark, we identify neurons of culture selectivity and perform diagnostic tests by deactivating the neurons flagged by various identification methods. Experiments on three VLMs across 25 cultural groups demonstrate the existence of neurons whose ablation disproportionately harms performance on questions about the corresponding cultures, while having limited effects on others. Moreover, we introduce a new margin-based selector Contrastive Activation Margin (ConAct) and show that it outperforms probability- and entropy-based methods in identifying neurons associated with cultural selectivity. Finally, our layer-wise analyses reveal that such neurons are not uniformly distributed: they cluster in specific decoder layers in a model-dependent way.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 22 of its 36 references that are in the map. First posted on arXiv 28 Oct 2025.

Builds on

22 of 36 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Grounding Multilingual Multimodal LLMs With Cultural Knowledge10 Aug
  2. 2024Benchmarking Vision Language Models for Cultural Understanding15 Jul
  3. 2024See It from My Perspective: How Language Affects Cultural Bias in Image Understanding17 Jun
  4. 2024CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark10 Jun

Other references · 18

  1. 2020GLU Variants Improve Transformercited by 1,724
  2. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  3. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,009
  4. 2023Steering Language Models With Activation Engineeringcited by 824
  5. 2017Network Dissection: Quantifying Interpretability of Deep Visual Representationscited by 703
  6. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 454
  7. 2020Understanding the Role of Individual Units in a Deep Neural Networkcited by 254
  8. 2021Composable Sparse Fine-Tuning for Cross-Lingual Transfercited by 144
  9. 2018What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Modelscited by 124
  10. 2018Identifying and Controlling Important Neurons in Neural Machine Translationcited by 110
  11. 2024Survey of Cultural Awareness in Language Models: Text and Beyondcited by 101
  12. 2024Universal Neurons in GPT2 Language Modelscited by 100
6 more
  1. 2023Neurons in Large Language Models: Dead, N-gram, Positionalcited by 88
  2. 2024Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languagescited by 63
  3. 2024Style-Specific Neurons for Steering LLMs in Text Style Transfercited by 24
  4. 2024Measuring the Inconsistency of Large Language Models in Preferential Rankingcited by 18
  5. 2025Beyond Words: Exploring Cultural Value Sensitivity in Multimodal Modelscited by 14
  6. 2025Cultural Awareness in Vision-Language Models: A Cross-Country Explorationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Neuron-Level Emotion Control in Speech-Generative Large Audio-Language Modelscited by 2
  2. 2026Cultural Binding Heads in Language Modelscited by 0
  3. 2026Emotion Across Speech and Faces: Shared Affective Mechanisms in Multimodal Foundation Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025CultureVLM: Characterizing and Improving Cultural Understanding of Vision-Language Models for over 100 Countries
  2. 2024CROPE: Evaluating In-Context Adaptation of Vision and Language Models to Culture-Specific Concepts
  3. 2025BLEnD-Vis: Benchmarking Multimodal Cultural Understanding in Vision Language Models
  4. 2025World in a Frame: Understanding Culture Mixing as a New Challenge for Vision-Language Models
  5. 2024Evaluating Visual and Cultural Interpretation: The K-Viscuit Benchmark with Human-VLM Collaboration
  6. 2025Uncovering Cultural Representation Disparities in Vision-Language Models
  7. 2025Seeing Culture: A Benchmark for Visual Reasoning and Grounding
  8. 2024From Local Concepts to Universals: Evaluating the Multicultural Understanding of Vision-Language Models