mlsanityData to 14 Oct 2026

Mixture of Channel Experts: Static Sparse Supports with Input-Adaptive Mixing for Pointwise Projections

arXiv 2608.23794 ↗First posted 24 Aug 2026cs.LG cs.CVcited by 0

On the map: Mixture-of-experts → Vision mixture-of-experts

Abstract (arXiv)

Mixture-of-Experts (MoE) scales language models by routing each input through a small set of independently parameterized experts. We show that copying this design into convolutional networks fails for a structural reason: parallel convolutional experts that read the same input channels learn nearly identical filters. We therefore move the expert axis from operator duplication to channel selection. We introduce Mixture of Channel Experts (MoCE), a structured sparse channel-mixing layer, inspired by MoE, that replaces pointwise (1x1) channel-reduction projections. In MoCE, an expert is a single output channel with a learned sparse support of k << C input channels. The selected channels are combined by a softmax whose temperature is predicted per input, so each expert can move between mean-like and max-like aggregation. A residual expert summarizes the unselected channels, and a load-balancing loss keeps channel coverage complete. MoCE replaces a dense projection whose cost is quadratic in C with a mechanism whose relative cost scales as k/C, and the predicted savings hold in measured wall-clock time. Across ResNet backbones on ImageNet-1K and CIFAR-100, transfer learning, EfficientViT, and a strong modern training recipe, MoCE matches or exceeds dense baselines and prior channel-selection methods while reducing MACs by 16.7% and end-to-end latency.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Vision mixture-of-experts, one of 12 directions in Mixture-of-experts. Builds on 20 of its 25 references that are in the map. First posted on arXiv 24 Aug 2026.

Builds on

20 of 25 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Routers in Vision Mixture of Experts: An Empirical Study29 Jan
  2. 2023From Sparse to Soft Mixtures of Experts2 Aug
  3. 2021Scaling Vision with Sparse Mixture of Experts10 Jun

Other references · 17

  1. 2017Squeeze-and-Excitation Networkscited by 4,789
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  3. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  4. 2022Mixture-of-Experts with Expert Choice Routingcited by 623
  5. 2016Learning Structured Sparsity in Deep Neural Networkscited by 541
  6. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 403
  7. 2019Dynamic Convolution: Attention over Convolution Kernelscited by 356
  8. 2019CondConv: Conditionally Parameterized Convolutions for Efficient Inferencecited by 273
  9. 2023EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attentioncited by 172
  10. 2018Dynamic Channel Pruning: Feature Boosting and Suppressioncited by 129
  11. 2017CondenseNet: An Efficient DenseNet using Learned Group Convolutionscited by 122
  12. 2021DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learningcited by 115
5 more
  1. 2018Channel Gating Neural Networkscited by 80
  2. 2019Discovering Neural Wiringscited by 56
  3. 2020Dynamic Group Convolution for Accelerating Convolutional Neural Networkscited by 15
  4. 2025DeRS: Towards Extremely Efficient Upcycled Mixture-of-Experts Modelscited by 10
  5. 2024Pick-or-Mix: Dynamic Channel Sampling for ConvNetscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Efficient Quantization of Mixture-of-Experts with Theoretical Generalization Guarantees
  2. 2026The Rise of Sparse Mixture-of-Experts: A Survey from Algorithmic Foundations to Decentralized Architectures and Vertical Domain Applications
  3. 2026Variational Inference, Entropy, and Orthogonality: A Unified Theory of Mixture-of-Experts
  4. 2024A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts
  5. 2023Scaling Vision-Language Models with Sparse Mixture of Experts
  6. 2018Deep Mixture of Experts via Shallow Embedding
  7. 2026Design and Behavior of Sparse Mixture-of-Experts Layers in CNN-based Semantic Segmentation
  8. 2025Sparse Mixture-of-Experts for Multi-Channel Imaging: Are All Channel Interactions Required?