# Universal Approximation of Mean-Field Models via Transformers

arXiv: https://arxiv.org/abs/2410.16295 · HTML: https://mlsanity.com/paper/2410.16295/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 28 references that are in the map.
- First posted on arXiv 6 Oct 2024.

## Builds on (most cited first, 14 of 14)

- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Masked Label Prediction: Unified Message Passing Model for Semi-Supervised Classification](https://mlsanity.com/paper/2009.03509/) (2020)
- [Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit](https://mlsanity.com/paper/1902.06015/) (2019)
- [The Lipschitz Constant of Self-Attention](https://mlsanity.com/paper/2006.04710/) (2020)
- [A mathematical perspective on Transformers](https://mlsanity.com/paper/2312.10794/) (2023)
- [A Mathematical Theory of Attention](https://mlsanity.com/paper/2007.02876/) (2020)
- [Nonparametric inference of interaction laws in systems of agents from trajectory data](https://mlsanity.com/paper/1812.06003/) (2018)
- [Transformers are Universal In-context Learners](https://mlsanity.com/paper/2408.01367/) (2024)
- [Universal Approximation Under Constraints is Possible with Transformers](https://mlsanity.com/paper/2110.03303/) (2021)
- [Sumformer: Universal Approximation for Efficient Transformers](https://mlsanity.com/paper/2307.02301/) (2023)
- [Learning Mean-Field Equations from Particle Data Using WSINDy](https://mlsanity.com/paper/2110.07756/) (2021)
- [Mean-field neural networks: learning mappings on Wasserstein space](https://mlsanity.com/paper/2210.15179/) (2022)
- [Learning Theory for Inferring Interaction Kernels in Second-Order Interacting Agent Systems](https://mlsanity.com/paper/2010.03729/) (2020)
- [A Survey on Universal Approximation Theorems](https://mlsanity.com/paper/2407.12895/) (2024)

## Built on by (most cited first, 7 of 7)

- [OT-Transformer: A Continuous-time Transformer Architecture with Optimal Transport Regularization](https://mlsanity.com/paper/2501.18793/) (2025)
- [Transformers through the lens of support-preserving maps between measures](https://mlsanity.com/paper/2509.25611/) (2025)
- [DiScoFormer: Plug-In Density and Score Estimation with Transformers](https://mlsanity.com/paper/2511.05924/) (2025)
- [One Operator for Many Densities: Amortized Approximation of Conditioning by Neural Operators](https://mlsanity.com/paper/2605.06873/) (2026)
- [Function graph transformers universally approximate operators between function spaces](https://mlsanity.com/paper/2605.17968/) (2026)
- [Measure-to-measure Regression with Transformers](https://mlsanity.com/paper/2605.28075/) (2026)
- [Universal Approximation of Measure-to-Measure Operators by Pushforwards](https://mlsanity.com/paper/2609.35483/) (2026)

## Abstract (arXiv)

This paper investigates the use of transformers to approximate the mean-field dynamics of interacting particle systems exhibiting collective behavior. Such systems are fundamental in modeling phenomena across physics, biology, and engineering, including opinion formation, biological networks, and swarm robotics. The key characteristic of these systems is that the particles are indistinguishable, leading to permutation-equivariant dynamics. First, we empirically demonstrate that transformers are well-suited for approximating a variety of mean field models, including the Cucker-Smale model for flocking and milling, and the mean-field system for training two-layer neural networks. We validate our numerical experiments via mathematical theory. Specifically, we prove that if a finite-dimensional transformer effectively approximates the finite-dimensional vector field governing the particle system, then the L2 distance between the textitexpected transformer and the infinite-dimensional mean-field vector field can be uniformly bounded by a function of the number of particles observed during training. Leveraging this result, we establish theoretical bounds on the distance between the true mean-field dynamics and those obtained using the transformer.