mlsanityData to 7 Oct 2026

Detecting a Route Flip Is Easier Than Knowing Whether to Fix It: Causal Route-Mediated Damage in Quantized Mixture-of-Experts

arXiv 2608.11212 ↗First posted 14 Jul 2026cs.AI cs.CL cs.LGcited by 1

On the map: Mixture-of-experts → Expert routing strategies

also in 1 more
Abstract (arXiv)

Top-k Mixture-of-Experts (MoE) routing is discontinuous, so a deployment-motivated numerical disturbance -- simulated 4-bit KV-cache quantization read by a protected BF16 gate -- pushes tokens across decision boundaries and flips which experts fire. This paper proposes no new mitigation; it supplies a causal apparatus, empirical findings, and a detection-limit result. A four-run apparatus prices the route-mediated fraction (RMF) of quantization damage, a token-level attribution decomposes it by mechanism, and pre-registered probes carry the findings across three architectures. On OLMoE-1B-7B at 4-bit KV (pilot), about a third of the damage is routing-mediated: RMF ~ 0.31 (discovery 0.31 [0.20, 0.41]; process-replicated mean 0.313 +/- 0.020; pre-registered re-execution 0.231). The deployable router margin detects that a flip occurred (AUC 0.772) but cannot tell a harmful flip from a helpful one (at chance): among the tested local, inference-observable router statistics we find no predictor of a flip's loss sign above chance -- an empirical benefit-detection barrier bounding selective repair restricted to this feature family. The signed-flip tax and sign-inseparability carry cross-model; the clean-reference remedy's payout is architecture-modulated; a controlled same-checkpoint flag-swap re-scopes the gate's normalization convention to a damage-magnitude moderator, not a route-recoverability mechanism. A real int4 KV kernel yields a fraction compatible with the fake-quant dose curve but underpowered (95% CI [-0.111, 0.394] includes zero) -- ruling out gross disagreement, not an independent replication. Hypotheses, thresholds, and evaluations were pre-registered before measurement, with misses reported; a pre-registered held-out read replicates the partition and the near-cancelling tax out of sample, while the strict impossibility exclusion narrowly misses.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 38 of its 39 references that are in the map. First posted on arXiv 14 Jul 2026.

Builds on

38 of 39 references.

Closest ancestors · 8

Same direction, newest first

  1. 2026Geometric and Stochastic Analysis of Discontinuities in Sparse Mixture-of-Experts17 Jun
  2. 2026SoftMoE: Soft Differentiable Routing for Mixture-of-Experts in LLMs16 Jun
  3. 2026From Observation to Intervention: A Causal Audit of Expert Importance in Mixture-of-Experts Models9 Jun
  4. 2026Routers Learn the Geometry of Their Experts: Geometric Coupling in Sparse Mixture-of-Experts12 May
  5. 2026Routing-Free Mixture-of-Experts1 Apr
  6. 2026Is Retraining-Free Enough? The Necessity of Router Calibration for Efficient MoE Compression10 Feb
  7. 2022Mixture-of-Experts with Expert Choice Routing18 Feb
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 30

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  4. 2016Pointer Sentinel Mixture Modelscited by 2,992
  5. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  6. 2024Qwen2 Technical Reportcited by 2,076
  7. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  8. 2024Mixtral of Expertscited by 1,785
  9. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  10. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  11. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  12. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
18 more
  1. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  2. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  3. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 598
  4. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  5. 2021Causal Abstractions of Neural Networkscited by 401
  6. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  7. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  8. 2023Attribution Patching Outperforms Automated Circuit Discoverycited by 196
  9. 2020Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Biascited by 183
  10. 2023Localizing Model Behavior with Path Patchingcited by 182
  11. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMscited by 165
  12. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 155
  13. 2025KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inferencecited by 36
  14. 2025EAQuant: Enhancing Post-Training Quantization for MoE Models via Expert-Aware Optimizationcited by 9
  15. 2025MixKVQ: Query-Aware Mixed-Precision KV Cache Quantization for Long-Context Reasoningcited by 8
  16. 2026GEMQ: Global Expert-Level Mixed-Precision Quantization for MoE LLMscited by 5
  17. 2026Value-and-Structure Alignment for Routing-Consistent Quantization of Mixture-of-Experts Modelscited by 5
  18. 2025KVmix: Gradient-Based Layer Importance-Aware Mixed-Precision Quantization for KV Cachecited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

Similar, not cited

Close in wording, no citation either way

  1. 2026A Persistent State for Auditable Mixture-of-Experts Routing
  2. 2026Safety-Oriented Routing Analysis of Mixtral MoE Under Benign and Harmful Prompts
  3. 2026Routing Drift Alone Does Not Diagnose Failure in Merged MoE LLMs
  4. 2026When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Models
  5. 2026Route-Block Membership Selects Packed-AWQ Arithmetic: A Controlled Single-Fixture Mechanism Study
  6. 2026Adaptive Inverted-Index Routing for Granular Mixtures-of-Experts
  7. 2026Router Sensitivity Under Lightweight Fine-Tuning Identifies Prunable Experts in Mixture-of-Experts Models
  8. 2026Router Prior Bias: Preserving Base Routing Structure in MoE Post-Training