# Anchor-MoE: A Mean-Anchored Mixture of Experts For Probabilistic Regression

arXiv: https://arxiv.org/abs/2508.16802 · HTML: https://mlsanity.com/paper/2508.16802/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts.
- Builds on 10 of its 19 references that are in the map.
- First posted on arXiv 22 Aug 2025.

## Placed in

- Mixture-of-experts / [MoE theory and gating functions](https://mlsanity.com/moe/moe-theory-and-gating-functions/)

## Builds on (most cited first, 10 of 10)

- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks](https://mlsanity.com/paper/1502.05336/) (2015)
- [Concrete Dropout](https://mlsanity.com/paper/1705.07832/) (2017)
- [NGBoost: Natural Gradient Boosting for Probabilistic Prediction](https://mlsanity.com/paper/1910.03225/) (2019)
- [Semi-Structured Distributional Regression -- Extending Structured Additive Models by Arbitrary Deep Neural Networks and Data Modalities](https://mlsanity.com/paper/2002.05777/) (2020)
- [Deconditional Downscaling with Gaussian Processes](https://mlsanity.com/paper/2105.12909/) (2021)
- [Multivariate Probabilistic Regression with Natural Gradient Boosting](https://mlsanity.com/paper/2106.03823/) (2021)
- [Ensembles of Probabilistic Regression Trees](https://mlsanity.com/paper/2406.14033/) (2024)

## Abstract (arXiv)

Regression under uncertainty is fundamental across science and engineering. We present an Anchored Mixture of Experts (Anchor-MoE), a model that handles both probabilistic and point regression. For simplicity, we use a tuned gradient-boosting model to furnish the anchor mean; however, any off-the-shelf point regressor can serve as the anchor. The anchor prediction is projected into a latent space, where a learnable metric-window kernel scores locality and a soft router dispatches each sample to a small set of mixture-density-network experts; the experts produce a heteroscedastic correction and predictive variance. We train by minimizing negative log-likelihood, and on a disjoint calibration split fit a post-hoc linear map on predicted means to improve point accuracy. On the theory side, assuming a H"older smooth regression function of order~α and fixed Lipschitz partition-of-unity weights with bounded overlap, we show that Anchor-MoE attains the minimax-optimal L2 risk rate O!big(N-2α/(2α+d)big). In addition, the CRPS test generalization gap scales as widetildeO!Big(√(log(Mh)+P+K)/NBig); it is logarithmic in Mh and scales as the square root in P and K. Under bounded-overlap routing, K can be replaced by k, and any dependence on a latent dimension is absorbed into P. Under uniformly bounded means and variances, an analogous widetildeO!big(√(log(Mh)+P+K)/Nbig) scaling holds for the test NLL up to constants. Empirically, across standard UCI regressions, Anchor-MoE consistently matches or surpasses the strong NGBoost baseline in RMSE and NLL; on several datasets it achieves new state-of-the-art probabilistic regression results on our benchmark suite. Code is available at https://github.com/BaozhuoSU/ProbabilisticRegression.