mlsanityData to 14 Oct 2026

Tracing Stereotypes from Representation to Output in Multilingual LLMs

arXiv 2609.08322 ↗First posted 8 Sep 2026cs.CL cs.AIcited by 0

On the map: Mechanistic interpretability → Probing deception, bias and beliefs

Abstract (arXiv)

Multilingual LLMs show stereotype-related behavior that varies across languages, but behavioral scores do not show where the relevant information is represented or how it affects the output. To investigate these internal mechanisms, we compare linear probing, attribution patching, sparse autoencoders (SAEs) and feature ablation in Llama-3.1-8B, Qwen3-8B, and Gemma-2-9B. Probe performance peaks substantially earlier than attribution in all three models, with a separation of 36-53% of model depth. Retained Llama-Scope features often match the social category on which they were selected and form recurring semantic families, but their lexical alignment and ablation effects vary across SAE suites. Only 6-18% of evaluated residual-stream features have language-agnostic effects under our criterion, and none are category-agnostic. Language-agnostic features have larger mean ablation effects in Llama-Scope, but this pattern does not repeat in the other SAE suites. Decodability, output influence, and cross-lingual ablation effects therefore need to be measured separately.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Probing deception, bias and beliefs, one of 13 directions in Mechanistic interpretability. Builds on 24 of its 39 references that are in the map. First posted on arXiv 8 Sep 2026.

Builds on

24 of 39 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Mechanistic Interpretability with SAEs: Probing Religion, Violence, and Geography in Large Language Models22 Sep

Other references · 23

  1. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  2. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  3. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  4. 2016Understanding intermediate layers using linear classifier probescited by 1,353
  5. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 710
  6. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 701
  7. 2019Designing and Interpreting Probes with Control Taskscited by 640
  8. 2024OLMo: Accelerating the Science of Language Modelscited by 613
  9. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 532
  10. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 394
  11. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 281
  12. 2023Attribution Patching Outperforms Automated Circuit Discoverycited by 209
11 more
  1. 2020Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance?cited by 151
  2. 2024Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoderscited by 143
  3. 2024RAVEL: Evaluating Interpretability Methods on Disentangling Language Model Representationscited by 79
  4. 2025SAEs Are Good for Steering -- If You Select the Right Featurescited by 78
  5. 2024CausalGym: Benchmarking causal interpretability methods on linguistic taskscited by 46
  6. 2024Separating Tongue from Thought: Activation Patching Reveals Language-Agnostic Concept Representations in Transformerscited by 44
  7. 2025Lost in Multilinguality: Dissecting Cross-lingual Factual Inconsistency in Transformer Language Modelscited by 44
  8. 2024MBBQ: A Dataset for Cross-Lingual Comparison of Stereotypes in Generative LLMscited by 35
  9. 2024SeeGULL Multilingual: a Dataset of Geo-Culturally Situated Stereotypescited by 24
  10. 2025Reading Between the Prompts: How Stereotypes Shape LLM's Implicit Personalizationcited by 24
  11. 2026Multilingual Steering by Design: Multilingual Sparse Autoencoders and Principled Layer Selectioncited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023What Do Llamas Really Think? Revealing Preference Biases in Language Model Representations
  2. 2024Understanding the role of FFNs in driving multilingual behaviour in LLMs
  3. 2024Interpreting Bias in Large Language Models: A Feature-Based Approach
  4. 2026Lingua Franca or Probing Artifact? Rethinking Latent Language in Multilingual LLMs
  5. 2026Cross-Lingual Bias in Large Language Models: A Comparative Analysis of English and Swahili
  6. 2025Tracing Multilingual Representations in LLMs with Cross-Layer Transcoders
  7. 2026The Interlingua Hypothesis: LLMs Translate via a Latent Task-agnostic Feature Space
  8. 2023Gender bias and stereotypes in Large Language Models