mlsanityData to 7 Oct 2026

L2R: Low-Rank and Lipschitz-Controlled Routing for Mixture-of-Experts

arXiv 2601.21349 ↗First posted 29 Jan 2026cs.LG cs.AIcited by 1

On the map: Mixture-of-experts → Vision mixture-of-experts

Abstract (arXiv)

Mixture-of-Experts (MoE) models scale neural networks by conditionally activating a small subset of experts, where the router plays a central role in determining expert specialization and overall model performance. However, many modern MoE systems still adopt linear routers in raw high-dimensional representation spaces, where representation mismatch, angular concentration, and scale-sensitive scoring can jointly undermine routing discriminability and stable expert specialization. In this work, we propose Low-rank & Lipschitz-controlled Routing (L2R), a unified routing framework that reshapes both the routing space and scoring geometry. L2R performs expert assignment in a shared low-rank latent routing space and introduces Saturated Inner-Product Scoring (SIPS) to explicitly control the Lipschitz behavior of routing functions, yielding smoother and more stable routing geometry. In addition, L2R incorporates a parameter-efficient multi-anchor routing mechanism to enhance expert expressiveness. Experiments on an OLMoE-based language MoE model and a ViT-based ImageNet setting show improved overall performance in both domains; OLMoE diagnostics further show improved routing geometry and expert discrimination.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Vision mixture-of-experts, one of 12 directions in Mixture-of-experts. Builds on 30 of its 37 references that are in the map. First posted on arXiv 29 Jan 2026.

Builds on

30 of 37 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Scaling Vision with Sparse Mixture of Experts10 Jun

Other references · 29

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  9. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  10. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  11. 2024Qwen2 Technical Reportcited by 2,076
  12. 2024Mixtral of Expertscited by 1,785
17 more
  1. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  2. 2019Root Mean Square Layer Normalizationcited by 1,425
  3. 2022Toy Models of Superpositioncited by 928
  4. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  5. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  6. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  7. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  8. 2024A Survey on Mixture of Experts in Large Language Modelscited by 233
  9. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 204
  10. 2022Tutel: Adaptive Mixture-of-Experts at Scalecited by 199
  11. 2022On the Representation Collapse of Sparse Mixture of Expertscited by 156
  12. 2024OLMES: A Standard for Language Model Evaluationscited by 100
  13. 2024Multi-Head Mixture-of-Expertscited by 30
  14. 2024Yuan 2.0-M32: Mixture of Experts with Attention Routercited by 18
  15. 2024Statistical Advantages of Perturbing Cosine Router in Mixture of Expertscited by 16
  16. 2025Adaptive Shared Experts with LoRA-Based Mixture of Experts for Multi-Task Learningcited by 3
  17. 2025Router Upcycling: Leveraging Mixture-of-Routers in Mixture-of-Experts Upcyclingcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026MoRE: Scaling mixture of experts with hardware-aware low-rank routingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Routers in Vision Mixture of Experts: An Empirical Study
  2. 2025Input Domain Aware MoE: Decoupling Routing Decisions from Task Optimization in Mixture of Experts
  3. 2026Self-Routing: Parameter-Free Expert Routing from Hidden States
  4. 2025Guided by the Experts: Provable Feature Learning Dynamic of Soft-Routed Mixture-of-Experts
  5. 2026Towards a Statistical Understanding of Mixture-of-Experts
  6. 2026The Rise of Sparse Mixture-of-Experts: A Survey from Algorithmic Foundations to Decentralized Architectures and Vertical Domain Applications
  7. 2024Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization
  8. 2026Path-Constrained Mixture-of-Experts