# Neural Scaling Laws of Transformer Operator Network

arXiv: https://arxiv.org/abs/2609.33533 · HTML: https://mlsanity.com/paper/2609.33533/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 35 references that are in the map.
- First posted on arXiv 27 Sep 2026.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operators](https://mlsanity.com/paper/2202.11214/) (2022)
- [Choose a Transformer: Fourier or Galerkin](https://mlsanity.com/paper/2105.14995/) (2021)
- [GNOT: A General Neural Operator Transformer for Operator Learning](https://mlsanity.com/paper/2302.14376/) (2023)
- [Transolver: A Fast Transformer Solver for PDEs on General Geometries](https://mlsanity.com/paper/2402.02366/) (2024)
- [Geometry-Informed Neural Operator for Large-Scale 3D PDEs](https://mlsanity.com/paper/2309.00583/) (2023)
- [Transformer for Partial Differential Equations' Operator Learning](https://mlsanity.com/paper/2205.13671/) (2022)
- [Inductive Biases and Variable Creation in Self-Attention Mechanisms](https://mlsanity.com/paper/2110.10090/) (2021)
- [Operator Learning: Algorithms and Analysis](https://mlsanity.com/paper/2402.15715/) (2024)
- [Deep Nonparametric Estimation of Operators between Infinite Dimensional Spaces](https://mlsanity.com/paper/2201.00217/) (2022)
- [Continuum Attention for Neural Operators](https://mlsanity.com/paper/2406.06486/) (2024)
- [Transformers as Neural Operators for Solutions of Differential Equations with Finite Regularity](https://mlsanity.com/paper/2405.19166/) (2024)
- [Understanding Scaling Laws with Statistical and Approximation Theory for Transformer Neural Networks on Intrinsically Low-dimensional Data](https://mlsanity.com/paper/2411.06646/) (2024)
- [The Parametric Complexity of Operator Learning](https://mlsanity.com/paper/2306.15924/) (2023)
- [HAMLET: Graph Transformer Neural Operator for Partial Differential Equations](https://mlsanity.com/paper/2402.03541/) (2024)
- [Positional Knowledge is All You Need: Position-induced Transformer (PiT) for Operator Learning](https://mlsanity.com/paper/2405.09285/) (2024)
- [Neural Scaling Laws of Deep ReLU and Deep Operator Network: A Theoretical Study](https://mlsanity.com/paper/2410.00357/) (2024)
- [Generalization Error Guaranteed Auto-Encoder-Based Nonlinear Model Reduction for Operator Learning](https://mlsanity.com/paper/2401.10490/) (2024)
- [Hierarchical Neural Operator Transformer with Learnable Frequency-aware Loss Prior for Arbitrary-scale Super-resolution](https://mlsanity.com/paper/2405.12202/) (2024)
- [Nonlinear functional regression by functional deep neural network with kernel embedding](https://mlsanity.com/paper/2401.02890/) (2024)
- [Generalization Guarantees for Multi-Input Neural Operator Learning in Sobolev Spaces](https://mlsanity.com/paper/2606.17419/) (2026)
- [Universal Approximation of Operators with Transformers and Neural Integral Operators](https://mlsanity.com/paper/2409.00841/) (2024)
- [Learning Theory of Transformers: Local-to-Global Approximation via Softmax Partition of Unity](https://mlsanity.com/paper/2605.08811/) (2026)
- [Function graph transformers universally approximate operators between function spaces](https://mlsanity.com/paper/2605.17968/) (2026)
- [Transformers as Cross-Task Learners: Shared Structure Drives Sample Efficiency in In-Context Learning](https://mlsanity.com/paper/2609.29060/) (2026)

## Abstract (arXiv)

Transformers have emerged as powerful architectures for learning solution operators of physical systems. Empirically the prediction error has been observed to decrease when the data size and model size increase, suggesting neural scaling behavior. Yet a theoretical understanding of such scaling laws for transformer-based operator learning remains limited. In this work, we develop a theoretical framework for characterizing the approximation and generalization errors of transformer-based operator learning. Our analysis builds on a local-to-global approximation principle that is naturally aligned with the softmax attention mechanism and yields discretization-invariant output functions. On approximation theory, we derive a universal approximation error of transformer-based operator learning for H"older-regular operators. On generalization theory, we establish a power scaling law between the prediction error and the training data size. The rate of convergence represented by the scaling exponent explicitly reflects the dimensions of the input and output domains, the regularity of the underlying functions and operators, and crucially, the intrinsic dimension of the input function class. By exploiting this intrinsic low-dimensional structure, our analysis yields a power-law generalization rate for operator learning, in contrast to the logarithmic-type power-law rates appearing in existing analyses of operator learning with feedforward neural networks. Numerical experiments validate the predicted power-law scaling and confirm that the convergence rate varies systematically with the intrinsic dimension of the input function class.