mlsanityData to 7 Oct 2026

Mixtures of Neural Operators Reduce Active Complexity in Operator Learning

arXiv 2404.09101 ↗First posted 13 Apr 2024cs.LG cs.AI cs.NAcited by 16

Not on the map. Why?

Abstract (arXiv)

Operator-learning systems are not governed solely by total parameter count; for one query, the relevant bottleneck can be the model that must be loaded and evaluated. We study this distinction for classical neural operators on compact Sobolev subsets through a constructive comparison between routed mixtures of neural operators (MoNOs) and a fixed single-neural-operator construction. The comparison concerns expert-active complexity relative to that baseline, with total stored size and routing search accounted separately. A MoNO routes each input function through a tree to one expert. Our main theorem shows that every scalar uniformly continuous nonlinear operator with bounded output Sobolev radius on the approximation set admits a MoNO approximation whose active expert has smaller depth, width, and rank scaling than the analyzed single-neural-operator construction; for Lipschitz targets these expert quantities are bounded by O(varepsilon-1). The theorem turns localization into an operator-level accounting of active expert size, routing depth, and number of experts. We also prove a quantitative universal approximation theorem for the underlying neural-operator architecture, with explicit dependence on compact-set diameter and modulus of continuity.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 29 of its 47 references that are in the map. First posted on arXiv 13 Apr 2024.

Builds on

29 of 47 references.

References · 29

  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  3. 2020Fourier Neural Operator for Parametric Partial Differential Equationscited by 2,806
  4. 2019DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operatorscited by 1,965
  5. 2024Mixtral of Expertscited by 1,785
  6. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  7. 2023Convolutional Neural Operators for robust and accurate learning of PDEscited by 207
  8. 2017Optimal approximation of piecewise smooth functions using deep ReLU neural networkscited by 176
  9. 2018Optimal approximation of continuous functions by very deep ReLU networkscited by 165
  10. 2019The phase diagram of approximation rates for deep neural networkscited by 86
  11. 2022Pathways: Asynchronous Distributed Dataflow for MLcited by 81
  12. 2023Neural Inverse Operators for Solving PDE Inverse Problemscited by 61
17 more
  1. 2023Representation Equivalent Neural Operators: a Framework for Alias-free Operator Learningcited by 57
  2. 2023Towards Stability of Autoregressive Neural Operatorscited by 54
  3. 2023Kernel Methods are Competitive for Operator Learningcited by 50
  4. 2021Exponential Convergence of Deep Operator Networks for Elliptic Partial Differential Equationscited by 44
  5. 2023Operator learning with PCA-Net: upper and lower complexity boundscited by 44
  6. 2021Universal Approximation Theorems for Differentiable Geometric Deep Learningcited by 41
  7. 2021Elementary superexpressive activationscited by 35
  8. 2023Deep Network Approximation: Beyond ReLU to Diverse Activation Functionscited by 33
  9. 2023The Parametric Complexity of Operator Learningcited by 27
  10. 2021Two-layer neural networks with values in a Banach spacecited by 26
  11. 2022Designing Universal Causal Deep Learning Models: The Geometric (Hyper)Transformercited by 22
  12. 2022Designing Universal Causal Deep Learning Models: The Case of Infinite-Dimensional Dynamical Systems from Stochastic Analysiscited by 22
  13. 2022Neural and spectral operator surrogates: unified construction and expression rate boundscited by 21
  14. 2019Deep learning architectures for nonlinear operator functions and nonlinear inverse problemscited by 17
  15. 2023Neural Oscillators are Universalcited by 16
  16. 2023Globally injective and bijective neural operatorscited by 13
  17. 2021Deep Neural Networks with ReLU-Sine-Exponential Activations Break Curse of Dimensionality in Approximation on H"older Classcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 6; 2026: 6.

16 papers.

Sort:

By area:

  1. 2024Operator Learning of Lipschitz Operators: An Information-Theoretic Perspectivecited by 9
  2. 2024Quantitative Approximation for Neural Operators in Nonlinear Parabolic Equationscited by 5
  3. 2024Neural Operators Can Play Dynamic Stackelberg Gamescited by 5
  4. 2025Generative Neural Operators of Log-Complexity Can Simultaneously Solve Infinitely Many Convex Programscited by 5
  5. 2025Verification and Validation for Trustworthy Scientific Machine Learningcited by 4
  6. 2024Low-dimensional approximations of the conditional law of Volterra processes: a non-positive curvature approachcited by 3
  7. 2025One model to solve them all: 2BSDE families via neural operatorscited by 3
  8. 2025On shallow feedforward neural networks with inputs from a topological spacecited by 2
  9. 2025Guided by the Experts: Provable Feature Learning Dynamic of Soft-Routed Mixture-of-Expertscited by 1
  10. 2025Upper Approximation Bounds for Neural Oscillatorscited by 1
  11. 2026Hybrid operator learning of wave scattering maps in high-contrast mediacited by 1
  12. 2026Universality of shallow and deep neural networks on non-Euclidean spacescited by 0
4 more
  1. 2026Neural Operators Can Discover Functional Clusterscited by 0
  2. 2026Structure-Preserving Reconstruction of Convex Lipschitz Functionals on Hilbert Spaces from Finite Samplescited by 0
  3. 2026Approximation Theory of Laplacian-Based Neural Operators for Reaction-Diffusion Systemcited by 0
  4. 2026Selection, Representation, and Execution in Sparse Fourier Neural Operatorscited by 0