mlsanityData to 7 Oct 2026

Systematic Exploration of 4-Expert Heterogeneous Mixture-of-Experts via Automated Pipeline Search

arXiv 2606.23739 ↗First posted 21 Jun 2026cs.LG cs.CV cs.SEcited by 1

On the map: Mixture-of-experts → MoE theory and gating functions

Abstract (arXiv)

We present an automated large-scale search pipeline for heterogeneous 4-Expert Mixture-of-Experts (MoE4) architectures within the LEMUR neural network dataset ecosystem. Building on a hand-crafted heterogeneous MoE reference model, we replace manual design with a deterministic code-assembly generator that systematically combines base architecture families drawn from the LEMUR database into MoE4 ensembles, each governed by a convolutional gating network with temperature scaling, mixup augmentation, and cosine-annealed learning rate scheduling. Over a 28-day campaign on an NVIDIA RTX 4090, the pipeline generated 4,463 candidate models across 197 batches, of which 1,021 were evaluated successfully. A critical finding emerged from the campaign: due to alphabetical enumeration via itertools.combinations, the entire explored search space (4.8% of the theoretical 23,751 possible 4-family combinations) is anchored to a single family, AirNet. We characterise this coverage bias precisely, identify the root cause in the generator, and propose a stratified random sampling fix. Within the AirNet anchored scope, ShuffleNet and MobileNetV3 consistently co-produce the highest-accuracy ensembles (mean accuracy up to 0.632), while FractalNet and MNASNet are identified as low-yield families warranting exclusion in future campaigns. The pipeline, analysis artefacts, and corrected generator are released as part of the open-source NNGPT project at https://github.com/ABrain-One/nn-gpt

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts. Builds on 24 of its 28 references that are in the map. First posted on arXiv 21 Jun 2026.

Builds on

24 of 28 references.

References · 24

  1. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  2. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  3. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  4. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 902
  5. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  6. 2019CondConv: Conditionally Parameterized Convolutions for Efficient Inferencecited by 273
  7. 2019Random Search and Reproducibility for Neural Architecture Searchcited by 253
  8. 2017Hard Mixtures of Experts for Large Scale Weakly Supervised Visioncited by 67
  9. 2016Network of Experts for Large-Scale Image Categorizationcited by 54
  10. 2025Optuna vs Code Llama: Are LLMs a New Paradigm for Hyperparameter Tuning?cited by 21
  11. 2025LEMUR Neural Network Dataset: Towards Seamless AutoMLcited by 20
  12. 2025NNGPT: Rethinking AutoML with Large Language Modelscited by 13
12 more
  1. 2025Preparation of Fractal-Inspired Computational Architectures for Advanced Large Language Model Analysiscited by 11
  2. 2025A Retrieval-Augmented Generation Approach to Extracting Algorithmic Logic from Neural Networkscited by 11
  3. 2025Enhancing LLM-Based Neural Network Generation: Few-Shot Prompting and Efficient Validation for Automated Architecture Designcited by 10
  4. 2026From Memorization to Creativity: LLM as a Designer of Novel Neural Architecturescited by 10
  5. 2025LLM as a Neural Architect: Controlled Generation of Image Captioning Models Under Strict API Contractscited by 9
  6. 2026From Brute Force to Semantic Insight: Performance-Guided Data Transformation Design with LLMscited by 7
  7. 2026Resource-Efficient Iterative LLM-Based NAS with Feedback Memorycited by 7
  8. 2026Closed-Loop LLM Discovery of Non-Standard Channel Priors in Vision Modelscited by 6
  9. 2026Delta-Based Neural Architecture Search: LLM Fine-Tuning via Code Diffscited by 3
  10. 2026MobileAgeNet: Lightweight Facial Age Estimation for Mobile Deploymentcited by 2
  11. 2026Real Image Denoising with Knowledge Distillation for High-Performance Mobile NPUscited by 2
  12. 2026Convergence Theory for Iterative LLM-Based Neural Architecture Search: A Parametric Cross-Entropy Framework with Closed-Form Proxy Reliabilitycited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Structure Aware Neural Architecture Search for Mixture of Expertscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Beyond Benchmarks: Understanding Mixture-of-Experts Models through Internal Mechanisms
  2. 2025Symphony-MoE: Harmonizing Disparate Pre-trained Models into a Coherent Mixture-of-Experts
  3. 2026MoE Lens -- An Expert Is All You Need
  4. 2026Mixture-of-Parallelisms: Towards Memory-Efficient Training Stack for Mixture-of-Experts Models
  5. 2023Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning
  6. 2026MAPLE: MoE Adaptive Plug-and-play Layer-wise Expert allocation
  7. 2026Scalable Training of Mixture-of-Experts Models with Megatron Core
  8. 2023A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Training