Expert Selections In MoE Models Reveal (Almost) As Much As Text
On the map: Mixture-of-experts → Expert routing strategies
Abstract (arXiv)
We present a text-reconstruction attack on mixture-of-experts (MoE) language models that recovers tokens from expert selections alone. In MoE models, each token is routed to a subset of expert subnetworks; we show these routing decisions leak substantially more information than previously understood. Prior work using logistic regression achieves limited reconstruction; we show that a 3-layer MLP improves this to 63.1% top-1 accuracy, and that a transformer-based sequence decoder recovers 91.2% of tokens top-1 (94.8% top-10) on 32-token sequences from OpenWebText after training on 100M tokens. These results connect MoE routing to the broader literature on embedding inversion. We outline practical leakage scenarios (e.g., distributed inference and side channels) and show that adding noise reduces but does not eliminate reconstruction. Our findings suggest that expert selections in MoE deployments should be treated as sensitive as the underlying text.
Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 6 of its 11 references that are in the map. First posted on arXiv 4 Feb 2026.
Builds on
6 of 11 references.
Closest ancestors · 1
Same direction, newest first
- 2025MoEcho: Exploiting Side-Channel Attacks to Compromise User Privacy in Mixture-of-Experts LLMs20 Aug
Other references · 5
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
- 2024Mixtral of Expertscited by 1,785
- 2020Information Leakage in Embedding Modelscited by 199
- 2025Language Models are Injective and Hence Invertiblecited by 31
- 2025Universal Zero-shot Embedding Inversioncited by 8
Built on by
Citing papers per year: 2026: 6.
6 papers.
- 2026The Myth of Expert Specialization in MoEs: Why Routing Reflects Geometry, Not Necessarily Domain Expertisecited by 9
- 2026RouteScan: A Non-Intrusive Approach to Auditing MoE LLMs Safety via Expert Routing Telemetrycited by 2
- 2026Layer-wise MoE Routing Locality under Shared-Prefix Code Generation: Token-Identity Decomposition and Compile-Equivalent Fork Redundancycited by 1
- 2026Workload Identification with Physical Side Channels for AI Governancecited by 0
- 2026CipherGenome: Homomorphic Inference for Genomic Mixture-of-Expertscited by 0
- 2026When Routing Reveals Membership: Privacy Leakage from MoE Router Telemetrycited by 0
- 2026When Routing Reveals Membership: Privacy Leakage from MoE Router Telemetry7 Oct 2026
- 2026CipherGenome: Homomorphic Inference for Genomic Mixture-of-Experts27 Sep 2026
- 2026Workload Identification with Physical Side Channels for AI Governance31 Aug 2026
- 2026RouteScan: A Non-Intrusive Approach to Auditing MoE LLMs Safety via Expert Routing Telemetry24 May 2026
- 2026Layer-wise MoE Routing Locality under Shared-Prefix Code Generation: Token-Identity Decomposition and Compile-Equivalent Fork Redundancy19 Apr 2026
- 2026The Myth of Expert Specialization in MoEs: Why Routing Reflects Geometry, Not Necessarily Domain Expertise10 Apr 2026
Similar, not cited
Close in wording, no citation either way
- 2025Exploiting the Experts: Unauthorized Compression in MoE-LLMs
- 2026RARE: Decoupling Representation Steering from Expert Routing in Mixture-of-Experts Language Models
- 2025FLAME-MoE: A Transparent End-to-End Research Platform for Mixture-of-Experts Language Models
- 2024Your Mixture-of-Experts LLM Is Secretly an Embedding Model For Free
- 2026Scaling Multi-Node Mixture-of-Experts Inference Using Expert Activation Patterns
- 2026Cross-Entropy Guided Routing in Mixture-of-Experts Large Language Models
- 2024Unchosen Experts Can Contribute Too: Unleashing MoE Models' Power by Self-Contrast
- 2026It Takes a MAESTRO To Prune Bad Experts