# Hierarchical Routing Mixture of Experts

arXiv: https://arxiv.org/abs/1903.07756 · HTML: https://mlsanity.com/paper/1903.07756/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts.
- Builds on 3 of its 38 references that are in the map.
- First posted on arXiv 18 Mar 2019.

## Placed in

- Mixture-of-experts / [Expert routing strategies](https://mlsanity.com/moe/expert-routing-strategies/)

## Builds on (most cited first, 3 of 3)

- [Distributed Gaussian Processes](https://mlsanity.com/paper/1502.02843/) (2015)
- [Neural Regression Trees](https://mlsanity.com/paper/1810.00974/) (2018)
- [Learning Deep Mixtures of Gaussian Process Experts Using Sum-Product Networks](https://mlsanity.com/paper/1809.04400/) (2018)

## Built on by (most cited first, 5 of 5)

- [Intelligent AI Delegation](https://mlsanity.com/paper/2602.11865/) (2026)
- [Mo"ET: Mixture of Expert Trees and its Application to Verifiable Reinforcement Learning](https://mlsanity.com/paper/1906.06717/) (2019)
- [φ-Balancing for Mixture-of-Experts Training](https://mlsanity.com/paper/2605.15403/) (2026)
- [Hypernetwork-Driven Low-Rank Adaptation Across Attention Heads](https://mlsanity.com/paper/2510.04295/) (2025)
- [Dynamic sparsity in tree-structured feed-forward layers at scale](https://mlsanity.com/paper/2604.08565/) (2026)

## Abstract (arXiv)

In regression tasks the distribution of the data is often too complex to be fitted by a single model. In contrast, partition-based models are developed where data is divided and fitted by local models. These models partition the input space and do not leverage the input-output dependency of multimodal-distributed data, and strong local models are needed to make good predictions. Addressing these problems, we propose a binary tree-structured hierarchical routing mixture of experts (HRME) model that has classifiers as non-leaf node experts and simple regression models as leaf node experts. The classifier nodes jointly soft-partition the input-output space based on the natural separateness of multimodal data. This enables simple leaf experts to be effective for prediction. Further, we develop a probabilistic framework for the HRME model, and propose a recursive Expectation-Maximization (EM) based algorithm to learn both the tree structure and the expert models. Experiments on a collection of regression tasks validate the effectiveness of our method compared to a variety of other regression models.