mlsanityData to 7 Oct 2026

Ray-Tracing for Conditionally Activated Neural Networks

arXiv 2502.14788 ↗First posted 20 Feb 2025cs.LG cs.AIcited by 0

On the map: Mixture-of-experts → MoE theory and gating functions

Abstract (arXiv)

In this paper, we introduce a novel architecture for conditionally activated neural networks combining a hierarchical construction of multiple Mixture of Experts (MoEs) layers with a sampling mechanism that progressively converges to an optimized configuration of expert activation. This methodology enables the dynamic unfolding of the network's architecture, facilitating efficient path-specific training. Experimental results demonstrate that this approach achieves competitive accuracy compared to conventional baselines while significantly reducing the parameter count required for inference. Notably, this parameter reduction correlates with the complexity of the input patterns, a property naturally emerging from the network's operational dynamics without necessitating explicit auxiliary penalty functions.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts. Builds on 7 of its 10 references that are in the map. First posted on arXiv 20 Feb 2025.

Builds on

7 of 10 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Conditional computation in neural networks: principles and research trends12 Mar

Other references · 6

  1. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  3. 2021Dynamic Neural Networks: A Surveycited by 405
  4. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 397
  5. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 242
  6. 2024Mixture of A Million Expertscited by 70

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Mixture of Raytraced Experts
  2. 2026DynaMoE: Dynamic Token-Level Expert Activation with Layer-Wise Adaptive Capacity for Mixture-of-Experts Neural Networks
  3. 2025Guided by the Experts: Provable Feature Learning Dynamic of Soft-Routed Mixture-of-Experts
  4. 2025Analytical FFN-to-MoE Restructuring via Activation Pattern Analysis
  5. 2026Meta-Learning Where to Allocate Experts: Task-Conditioned Layer-Wise Compression for MoEs
  6. 2025Efficient Data Driven Mixture-of-Expert Extraction from Trained Networks
  7. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning
  8. 2026MoE Lens -- An Expert Is All You Need