mlsanityData to 7 Oct 2026

Knowledge Localization in Mixture-of-Experts LLMs Using Cross-Lingual Inconsistency

arXiv 2603.17102 ↗First posted 17 Mar 2026cs.CL cs.AI cs.LGcited by 6

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Modern LLMs continue to exhibit significant variance in behavior across languages, such as being able to recall factual information in some languages but not others. While typically studied as a problem to be mitigated, in this work, we propose leveraging this cross-lingual inconsistency as a tool for interpretability in mixture-of-experts (MoE) LLMs. Our knowledge localization framework contrasts routing for sets of languages where the model correctly recalls information from languages where it fails. This allows us to isolate model components that play a functional role in answering about a piece of knowledge. Our method proceeds in two stages: (1) querying the model with difficult factual questions across a diverse set of languages to generate "success" and "failure" activation buckets and then (2) applying a statistical contrastive analysis to the MoE router logits to identify experts important for knowledge. To validate the necessity of this small number of experts for answering a knowledge question, we deactivate them and re-ask the question. We find that despite only deactivating about 20 out of 6000 experts, the model no longer answers correctly in over 40% of cases. Generally, this method provides a realistic and scalable knowledge localization approach to address increasingly complex LLMs.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 38 of its 48 references that are in the map. First posted on arXiv 17 Mar 2026.

Builds on

38 of 48 references.

Closest ancestors · 5

Same direction, newest first

  1. 2026MoEEdit: Efficient and Routing-Stable Knowledge Editing for Mixture-of-Experts LLMs11 Feb
  2. 2026Deconstructing Pre-training: Knowledge Attribution Analysis in MoE and Dense Models13 Jan
  3. 2024A Closer Look into Mixture-of-Experts in Large Language Models26 Jun
  4. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models29 Jan
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 33

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  3. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  4. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  5. 2022Mass-Editing Memory in a Transformercited by 976
  6. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  7. 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 293
  8. 2023Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Modelscited by 237
  9. 2023PMET: Precise Model Editing in a Transformercited by 170
  10. 2023Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Modelscited by 117
  11. 2024On the Multilingual Ability of Decoder-based Pre-trained Language Models: Finding and Controlling Language-Specific Neuronscited by 83
  12. 2024INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledgecited by 78
21 more
  1. 2024The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalitiescited by 69
  2. 2022A Rigorous Study of Integrated Gradients Method and Extensions to Internal Neuron Attributionscited by 64
  3. 2025Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Modelscited by 53
  4. 2024Separating Tongue from Thought: Activation Patching Reveals Language-Agnostic Concept Representations in Transformerscited by 42
  5. 2023Do Localization Methods Actually Localize Memorized Data in LLMs? A Tale of Two Benchmarkscited by 29
  6. 2025ECLeKTic: a Novel Challenge Set for Evaluation of Cross-Lingual Knowledge Transfercited by 28
  7. 2025Multilingual Routing in Mixture-of-Expertscited by 25
  8. 2025Steering MoE LLMs via Expert (De)Activationcited by 23
  9. 2025MultiLoKo: a multilingual local knowledge benchmark for LLMs spanning 31 languagescited by 22
  10. 2025Language-Specific Latent Process Hinders Cross-Lingual Performancecited by 20
  11. 2025Paths Not Taken: Understanding and Mending the Multilingual Factual Recall Pipelinecited by 20
  12. 2025The Emergence of Abstract Thought in Large Language Models Beyond Any Languagecited by 20
  13. 2025Probing Semantic Routing in Large Mixture-of-Expert Modelscited by 13
  14. 2025Tracing Multilingual Factual Knowledge Acquisition in Pretrainingcited by 10
  15. 2024Does Knowledge Localization Hold True? Surprising Differences Between Entity and Relation Perspectives in Language Modelscited by 8
  16. 2025The Unreasonable Effectiveness of Model Merging for Cross-Lingual Transfer in LLMscited by 8
  17. 2025Quantifying Language Disparities in Multilingual Large Language Modelscited by 6
  18. 2025Beyond Benchmarks: Understanding Mixture-of-Experts Models through Internal Mechanismscited by 6
  19. 2025Analyzing LLMs' Knowledge Boundary Cognition Across Languages Through the Lens of Internal Representationscited by 5
  20. 2025Rethinking Cross-lingual Gaps from a Statistical Viewpointcited by 4
  21. 2025ACE: Attribution-Controlled Knowledge Editing for Multi-hop Factual Recallcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Extracting Small Translation Specialists from LLMs by Aggressively Pruning Expertscited by 3
  2. 2026Cross-Lingual Steering for Figurative Language Generationcited by 1
  3. 2026A theoretical model for task routing in mixture-of-expert transformerscited by 0
  4. 2026LMs as Task-Specific Knowledge Bases: An Interpretability Analysiscited by 0
  5. 2026Multilinguality in Hybrid Attention LLMscited by 0
  6. 2026Cross-Lingual Alignment for Decoder-Only Models using MoE Routerscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Understanding Multilingualism in Mixture-of-Experts LLMs: Routing Mechanism, Expert Specialization, and Layerwise Steering
  2. 2024A Survey on Mixture of Experts in Large Language Models
  3. 2025Mixture of Thoughts: Learning to Aggregate What Experts Think, Not Just What They Say
  4. 2025Evaluating Expert Contributions in a MoE LLM for Quiz-Based Tasks
  5. 2024Leveraging Open Knowledge for Advancing Task Expertise in Large Language Models
  6. 2025Bayesian Mixture-of-Experts: Towards Making LLMs Know What They Don't Know
  7. 2026Do Domain-specific Experts exist in MoE-based LLMs?
  8. 2026Mixture of Experts for Low-Resource LLMs