mlsanityData to 14 Oct 2026

Mechanistic Interpretability with SAEs: Probing Religion, Violence, and Geography in Large Language Models

arXiv 2509.17665 ↗First posted 22 Sep 2025cs.LG cs.AI cs.CYcited by 4

On the map: Mechanistic interpretability → Probing deception, bias and beliefs

Abstract (arXiv)

Despite growing research on bias in large language models (LLMs), most work has focused on gender and race, with little attention to religious identity. This paper explores how religion is internally represented in LLMs and how it intersects with concepts of violence and geography. Using mechanistic interpretability and Sparse Autoencoders (SAEs) via the Neuronpedia API, we analyze latent feature activations across five models. We measure overlap between religion- and violence-related prompts and probe semantic patterns in activation contexts. While all five religions show comparable internal cohesion, Islam is more frequently linked to features associated with violent language. In contrast, geographic associations largely reflect real-world religious demographics, revealing how models embed both factual distributions and cultural stereotypes. These findings highlight the value of structural analysis in auditing not just outputs but also internal representations that shape model behavior.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Probing deception, bias and beliefs, one of 13 directions in Mechanistic interpretability. Builds on 23 of its 27 references that are in the map. First posted on arXiv 22 Sep 2025.

Builds on

23 of 27 references.

References · 23

  1. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  2. 2019Attention is not Explanationcited by 929
  3. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 924
  4. 2016Semantics derived automatically from language corpora contain human-like biasescited by 901
  5. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 724
  6. 2023Bias and Fairness in Large Language Models: A Surveycited by 639
  7. 2019On Measuring Social Biases in Sentence Encoderscited by 398
  8. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 394
  9. 2021Persistent Anti-Muslim Bias in Large Language Modelscited by 377
  10. 2024Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoderscited by 300
  11. 2019A Multiscale Visualization of Attention in the Transformer Modelcited by 292
  12. 2019Measuring Bias in Contextualized Word Representationscited by 276
11 more
  1. 2023A Toy Model of Universality: Reverse Engineering How Networks Learn Group Operationscited by 144
  2. 2024Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoderscited by 143
  3. 2018What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Modelscited by 124
  4. 2021An Interpretability Illusion for BERTcited by 97
  5. 2023A Survey on Fairness in Large Language Modelscited by 95
  6. 2025A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Modelscited by 76
  7. 2021Unmasking the Mask -- Evaluating Social Biases in Masked Language Modelscited by 60
  8. 2025I Have Covered All the Bases Here: Interpreting Reasoning Features in Large Language Models via Sparse Autoencoderscited by 45
  9. 2023Large Language Model (LLM) Bias Index -- LLMBIcited by 14
  10. 2024Divine LLaMAs: Bias, Stereotypes, Stigmatization, and Emotion Representation of Religion in Large Language Modelscited by 12
  11. 2025Religious Bias Landscape in Language and Text-to-Image Models: Analysis, Detection, and Debiasing Strategiescited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2026Mechanistic Interpretability of Cognitive Complexity in LLMs via Linear Probing using Bloom's Taxonomycited by 2
  2. 2025GPT and Prejudice: A Sparse Approach to Understanding Learned Representations in Large Language Modelscited by 0
  3. 2026StereoTales: A Multilingual Framework for Open-Ended Stereotype Discovery in LLMscited by 0
  4. 2026Tracing Stereotypes from Representation to Output in Multilingual LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023What Do Llamas Really Think? Revealing Preference Biases in Language Model Representations
  2. 2025Surfacing Subtle Stereotypes: A Multilingual, Debate-Oriented Evaluation of Modern LLMs
  3. 2024Large Language Models are Geographically Biased
  4. 2025When Models Refuse: Political Steerability and Feature Richness as Measures of Ideological Depth
  5. 2026Tracing Moral Foundations in Large Language Models
  6. 2025Examining Alignment of Large Language Models through Representative Heuristics: The Case of Political Stereotypes
  7. 2026Political Alignment in Large Language Models: A Multidimensional Audit of Psychometric Identity and Behavioral Bias
  8. 2024Brittle Minds, Fixable Activations: Understanding Belief Representations in Language Models