Anchor-MoE: A Mean-Anchored Mixture of Experts For Probabilistic Regression

Regression under uncertainty is fundamental across science and engineering. We present an Anchored Mixture of Experts (Anchor-MoE), a model that handles both probabilistic and point regression. For simplicity, we use a tuned gradient-boosting model to furnish the anchor mean; however, any off-the-shelf point regressor can serve as the anchor. The anchor prediction is projected into a latent space, where a learnable metric-window kernel scores locality and a soft router dispatches each sample to a small set of mixture-density-network experts; the experts produce a heteroscedastic correction and predictive variance. We train by minimizing negative log-likelihood, and on a disjoint calibration split fit a post-hoc linear map on predicted means to improve point accuracy. On the theory side, assuming a H"older smooth regression function of order~α and fixed Lipschitz partition-of-unity weights with bounded overlap, we show that Anchor-MoE attains the minimax-optimal L2 risk rate O!big(N-2α/(2α+d)big). In addition, the CRPS test generalization gap scales as widetildeO!Big(√(log(Mh)+P+K)/NBig); it is logarithmic in Mh and scales as the square root in P and K. Under bounded-overlap routing, K can be replaced by k, and any dependence on a latent dimension is absorbed into P. Under uniformly bounded means and variances, an analogous widetildeO!big(√(log(Mh)+P+K)/Nbig) scaling holds for the test NLL up to constants. Empirically, across standard UCI regressions, Anchor-MoE consistently matches or surpasses the strong NGBoost baseline in RMSE and NLL; on several datasets it achieves new state-of-the-art probabilistic regression results on our benchmark suite. Code is available at https://github.com/BaozhuoSU/ProbabilisticRegression.