# Network of Experts for Large-Scale Image Categorization

arXiv: https://arxiv.org/abs/1604.06119 · HTML: https://mlsanity.com/paper/1604.06119/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Apr 2016.

## Built on by (most cited first, 30 of 54)

- [Scaling Vision with Sparse Mixture of Experts](https://mlsanity.com/paper/2106.05974/) (2021)
- [MOS: Towards Scaling Out-of-distribution Detection for Large Semantic Space](https://mlsanity.com/paper/2105.01879/) (2021)
- [M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-design](https://mlsanity.com/paper/2210.14793/) (2022)
- [Deep Hierarchical Semantic Segmentation](https://mlsanity.com/paper/2203.14335/) (2022)
- [Generalizable Person Re-identification with Relevance-aware Mixture of Experts](https://mlsanity.com/paper/2105.09156/) (2021)
- [Hard Mixtures of Experts for Large Scale Weakly Supervised Vision](https://mlsanity.com/paper/1704.06363/) (2017)
- [Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers](https://mlsanity.com/paper/2303.01610/) (2023)
- [Meta-DMoE: Adapting to Domain Shift by Meta-Distillation from Mixture-of-Experts](https://mlsanity.com/paper/2210.03885/) (2022)
- [SpeechMoE: Scaling to Large Acoustic Models with Dynamic Routing Mixture of Experts](https://mlsanity.com/paper/2105.03036/) (2021)
- [The Neural Race Reduction: Dynamics of Abstraction in Gated Networks](https://mlsanity.com/paper/2207.10430/) (2022)
- [Patch-level Routing in Mixture-of-Experts is Provably Sample-efficient for Convolutional Neural Networks](https://mlsanity.com/paper/2306.04073/) (2023)
- [Robust Mixture-of-Expert Training for Convolutional Neural Networks](https://mlsanity.com/paper/2308.10110/) (2023)
- [Logic-induced Diagnostic Reasoning for Semi-supervised Semantic Segmentation](https://mlsanity.com/paper/2308.12595/) (2023)
- [Adaptive Mixture of Experts Learning for Generalizable Face Anti-Spoofing](https://mlsanity.com/paper/2207.09868/) (2022)
- [Enhancing NeRF akin to Enhancing LLMs: Generalizable NeRF Transformer with Mixture-of-View-Experts](https://mlsanity.com/paper/2308.11793/) (2023)
- [SpeechMoE2: Mixture-of-Experts Model with Improved Routing](https://mlsanity.com/paper/2111.11831/) (2021)
- [Routers in Vision Mixture of Experts: An Empirical Study](https://mlsanity.com/paper/2401.15969/) (2024)
- [Psychometry: An Omnifit Model for Image Reconstruction from Human Brain Activity](https://mlsanity.com/paper/2403.20022/) (2024)
- [Neural Networks are Decision Trees](https://mlsanity.com/paper/2210.05189/) (2022)
- [HydraFusion: Context-Aware Selective Sensor Fusion for Robust and Efficient Autonomous Vehicle Perception](https://mlsanity.com/paper/2201.06644/) (2022)
- [Learning Topology-Specific Experts for Molecular Property Prediction](https://mlsanity.com/paper/2302.13693/) (2023)
- [Sparsely-gated Mixture-of-Expert Layers for CNN Interpretability](https://mlsanity.com/paper/2204.10598/) (2022)
- [Hierarchical classification at multiple operating points](https://mlsanity.com/paper/2210.10929/) (2022)
- [Spatial Mixture-of-Experts](https://mlsanity.com/paper/2211.13491/) (2022)
- [ProTeCt: Prompt Tuning for Taxonomic Open Set Classification](https://mlsanity.com/paper/2306.02240/) (2023)
- [Domain Generalization Using Large Pretrained Models with Mixture-of-Adapters](https://mlsanity.com/paper/2310.11031/) (2023)
- [Semi-Supervised Video Desnowing Network via Temporal Decoupling Experts and Distribution-Driven Contrastive Regularization](https://mlsanity.com/paper/2410.07901/) (2024)
- [Fine-grain Inference on Out-of-Distribution Data with Hierarchical Classification](https://mlsanity.com/paper/2209.04493/) (2022)
- [Revisiting Single-gated Mixtures of Experts](https://mlsanity.com/paper/2304.05497/) (2023)
- [HAMoBE: Hierarchical and Adaptive Mixture of Biometric Experts for Video-based Person ReID](https://mlsanity.com/paper/2508.05038/) (2025)

## Abstract (arXiv)

We present a tree-structured network architecture for large scale image classification. The trunk of the network contains convolutional layers optimized over all classes. At a given depth, the trunk splits into separate branches, each dedicated to discriminate a different subset of classes. Each branch acts as an expert classifying a set of categories that are difficult to tell apart, while the trunk provides common knowledge to all experts in the form of shared features. The training of our "network of experts" is completely end-to-end: the partition of categories into disjoint subsets is learned simultaneously with the parameters of the network trunk and the experts are trained jointly by minimizing a single learning objective over all classes. The proposed structure can be built from any existing convolutional neural network (CNN). We demonstrate its generality by adapting 4 popular CNNs for image categorization into the form of networks of experts. Our experiments on CIFAR100 and ImageNet show that in every case our method yields a substantial improvement in accuracy over the base CNN, and gives the best result achieved so far on CIFAR100. Finally, the improvement in accuracy comes at little additional cost: compared to the base network, the training time is only moderately increased and the number of parameters is comparable or in some cases even lower.