mlsanityData to 7 Oct 2026

Probabilistic partition of unity networks for high-dimensional regression problems

arXiv 2210.02694 ↗First posted 6 Oct 2022cs.LG cs.CEcited by 2

On the map: Mixture-of-experts → MoE theory and gating functions

Abstract (arXiv)

We explore the probabilistic partition of unity network (PPOU-Net) model in the context of high-dimensional regression problems and propose a general framework focusing on adaptive dimensionality reduction. With the proposed framework, the target function is approximated by a mixture of experts model on a low-dimensional manifold, where each cluster is associated with a local fixed-degree polynomial. We present a training strategy that leverages the expectation maximization (EM) algorithm. During the training, we alternate between (i) applying gradient descent to update the DNN coefficients; and (ii) using closed-form formulae derived from the EM algorithm to update the mixture of experts model parameters. Under the probabilistic formulation, step (ii) admits the form of embarrassingly parallelizable weighted least-squares solves. The PPOU-Nets consistently outperform the baseline fully-connected neural networks of comparable sizes in numerical experiments of various data dimensions. We also explore the proposed model in applications of quantum computing, where the PPOU-Nets act as surrogate models for cost landscapes associated with variational quantum circuits.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in MoE theory and gating functions, one of 12 directions in Mixture-of-experts. Builds on 32 of its 110 references that are in the map. First posted on arXiv 6 Oct 2022.

Builds on

32 of 110 references.

References · 32

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2013Auto-Encoding Variational Bayescited by 3,980
  4. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  5. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  6. 2019DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operatorscited by 1,965
  7. 2016Variational Inference: A Review for Statisticianscited by 1,466
  8. 2017Deep Image Priorcited by 1,097
  9. 2019An Introduction to Variational Autoencoderscited by 997
  10. 2015Automatic differentiation in machine learning: a surveycited by 846
  11. 2017DGM: A deep learning algorithm for solving partial differential equationscited by 674
  12. 2016Error bounds for approximations with deep ReLU networkscited by 567
20 more
  1. 2017Solving high-dimensional partial differential equations using deep learningcited by 549
  2. 2016From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classificationcited by 337
  3. 2014Breaking the Curse of Dimensionality with Convex Neural Networkscited by 329
  4. 2019Physics-Constrained Deep Learning for High-dimensional Surrogate Modeling and Uncertainty Quantification without Labeled Datacited by 261
  5. 2018Bayesian Deep Convolutional Encoder-Decoder Networks for Surrogate Modeling and Uncertainty Quantificationcited by 199
  6. 2018Optimal approximation of continuous functions by very deep ReLU networkscited by 165
  7. 2016Why and When Can Deep -- but Not Shallow -- Networks Avoid the Curse of Dimensionality: a Reviewcited by 155
  8. 2018Quantifying total uncertainty in physics-informed neural networks for solving forward and inverse stochastic problemscited by 144
  9. 2017A unified deep artificial neural network approach to partial differential equations in complex geometriescited by 122
  10. 2018Deep UQ: Learning deep neural network surrogate models for high dimensional uncertainty quantificationcited by 81
  11. 2014Manifold Gaussian Processes for Regressioncited by 79
  12. 2019Robust Training and Initialization of Deep Neural Networks: An Adaptive Basis Viewpointcited by 42
  13. 2019DPM: A deep learning PDE augmentation method (with application to large-eddy simulation)cited by 40
  14. 2019Chart Auto-Encoders for Manifold Structured Datacited by 32
  15. 2022A survey of unsupervised learning methods for high-dimensional uncertainty quantification in black-box-type problemscited by 20
  16. 2018Extending classical surrogate modelling to high-dimensions through supervised dimensionality reduction: a data-driven approachcited by 16
  17. 2021Partition of unity networks: deep hp-approximationcited by 15
  18. 2018Gradient-based Optimization for Regression in the Functional Tensor-Train Formatcited by 7
  19. 2016Filling the gaps: Gaussian mixture models from noisy, truncated or incomplete samplescited by 5
  20. 2021Probabilistic partition of unity networks: clustering based deep approximationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025The Longitudinal Health, Income, and Employment Model (LHIEM): a discrete-time microsimulation model for policy analysiscited by 0
  2. 2026From Classification to Regression: Using a Fruitfly to Solve Equationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Mean-field limit from general mixtures of experts to quantum neural networks
  2. 2022Parameter-Efficient Mixture-of-Experts Architecture for Pre-trained Language Models
  3. 2025Mixture of Many Zero-Compute Experts: A High-Rate Quantization Theory Perspective
  4. 2023Revisiting Single-gated Mixtures of Experts
  5. 2026TileQ: Efficient Low-Rank Quantization of Mixture-of-Experts with 2D Tiling
  6. 2026Grassmannian Mixture-of-Experts: Concentration-Controlled Routing on Subspace Manifolds
  7. 2024Gradient-free variational learning with conditional mixture networks
  8. 2026MoE-DisCo:Low Economy Cost Training Mixture-of-Experts Models