mlsanityData to 14 Oct 2026

Mixture of Raytraced Experts

arXiv 2507.12419 ↗First posted 16 Jul 2025cs.LG cs.AIcited by 0

On the map: Mixture-of-experts → MoE theory and gating functions

Abstract (arXiv)

We introduce a Mixture of Raytraced Experts, a stacked Mixture of Experts (MoE) architecture which can dynamically select sequences of experts, producing computational graphs of variable width and depth. Existing MoE architectures generally require a fixed amount of computation for a given sample. Our approach, in contrast, yields predictions with increasing accuracy as the computation cycles through the experts' sequence. We train our model by iteratively sampling from a set of candidate experts, unfolding the sequence akin to how Recurrent Neural Networks are trained. Our method does not require load-balancing mechanisms, and preliminary experiments show a reduction in training epochs of 10% to 40% with a comparable/higher accuracy. These results point to new research directions in the field of MoEs, allowing the design of potentially faster and more expressive models. The code is available at https://github.com/nutig/RayTracing

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts. Builds on 8 of its 18 references that are in the map. First posted on arXiv 16 Jul 2025.

Builds on

8 of 18 references.

References · 8

  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  3. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 596
  4. 2024Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Expertscited by 205
  5. 2024Harder Tasks Need More Experts: Dynamic Routing in MoE Modelscited by 72
  6. 2025Load Balancing Mixture of Experts with Similarity Preserving Routerscited by 22
  7. 2025Chain-of-Experts: Unlocking the Communication Power of Mixture-of-Experts Modelscited by 9
  8. 2025MoE-GPS: Guidlines for Prediction Strategy for Dynamic Expert Duplication in MoE Load Balancingcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Symphony-MoE: Harmonizing Disparate Pre-trained Models into a Coherent Mixture-of-Experts
  2. 2026∞-MoE: Generalizing Mixture of Experts to Infinite Experts
  3. 2023Improving Expert Specialization in Mixture of Experts
  4. 2023Revisiting Single-gated Mixtures of Experts
  5. 2025On the Expressive Power of Mixture-of-Experts for Structured Complex Tasks
  6. 2024SEER-MoE: Sparse Expert Efficiency through Regularization for Mixture-of-Experts
  7. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applications
  8. 2024Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization