mlsanityData to 14 Oct 2026

Expert-Aware Causal Tracing of Factual Recall in Sparse MoE Language Models

arXiv 2606.03780 ↗First posted 2 Jun 2026cs.CL cs.LGcited by 0

On the map: Mixture-of-experts → Expert routing strategies

also in 1 more
Abstract (arXiv)

Activation patching can identify a mixture-of-experts (MoE) block whose clean output restores a corrupted factual prediction. However, because the block output combines contributions from multiple routed experts, block-level rescue does not establish whether the recovery localizes to an individual expert or depends on the routed expert set. We study this question on single-token COUNTERFACT contrasts by corrupting subject-token embeddings, restoring clean block outputs, and then restoring clean-minus-noised expert updates under fixed routing. In Qwen3-30B-A3B-Base, a discovery sweep selects layer 44, and held-out analysis identifies L44E069 as a recurrent routed contributor with positive specificity over same-layer active experts. Its effect is fact-matched and improves true-token probability and rank, which explains part of the layer rescue. In Mixtral-8x7B-v0.1, the selected recurrent singleton is not specific; matched-size controls instead show specificity of the clean-routed top-2 set. These findings show that successful MoE-block restoration does not necessarily imply localization to a single expert.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 14 of its 24 references that are in the map. First posted on arXiv 2 Jun 2026.

Builds on

14 of 24 references.

Closest ancestors · 4

Same direction, newest first

  1. 2026The Myth of Expert Specialization in MoEs: Why Routing Reflects Geometry, Not Necessarily Domain Expertise10 Apr
  2. 2025Multilingual Routing in Mixture-of-Experts6 Oct
  3. 2025Steering MoE LLMs via Expert (De)Activation11 Sep
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 10

  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  2. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  3. 2019Language Models as Knowledge Bases?cited by 1,936
  4. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  5. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  6. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  7. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  8. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 528
  9. 2024Unveiling Factual Recall Behaviors of Large Language Models through Knowledge Neuronscited by 16
  10. 2026MoEEdit: Efficient and Routing-Stable Knowledge Editing for Mixture-of-Experts LLMscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Awakening Dormant Experts:Counterfactual Routing to Mitigate MoE Hallucinations
  2. 2026Expert Selections In MoE Models Reveal (Almost) As Much As Text
  3. 2026Cross-Entropy Guided Routing in Mixture-of-Experts Large Language Models
  4. 2026Routing-Aware Expert Calibration for Machine Unlearning in Mixture-of-Experts Language Models
  5. 2026THESIS-MoE: Trainable Hierarchical Extraction and SteerIng of Sycophancy in Mixture-of-Experts
  6. 2026It Takes a MAESTRO To Prune Bad Experts
  7. 2026Beyond Geometric Complementarity: Coherent Overlap in Sparse Mixture-of-Experts Routing
  8. 2026REFLEX: Rethinking MoE Inference as Refinement-Aware Compute Allocation in Diffusion Language Models