mlsanityData to 7 Oct 2026

Measure-to-measure interpolation using Transformers

arXiv 2411.04551 ↗First posted 7 Nov 2024math.OC cs.LG stat.MLcited by 28

Not on the map. Why?

Abstract (arXiv)

Transformers are deep neural network architectures that underpin the recent successes of large language models. Unlike more classical architectures that can be viewed as point-to-point maps, a Transformer acts as a measure-to-measure map implemented as specific interacting particle system on the unit sphere: the input is the empirical measure of tokens in a prompt and its evolution is governed by the continuity equation. In fact, Transformers are not limited to empirical measures and can in principle process any input measure. As the nature of data processed by Transformers is expanding rapidly, it is important to investigate their expressive power as maps from an arbitrary measure to another arbitrary measure. To that end, we provide an explicit choice of parameters that allows a single Transformer to match N arbitrary input measures to N arbitrary target measures, under the minimal assumption that every pair of input-target measures can be matched by some transport map.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 47 of its 76 references that are in the map. First posted on arXiv 7 Nov 2024.

Builds on

47 of 76 references.

References · 47

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  3. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 411
  4. 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 404
  5. 2023A mathematical perspective on Transformerscited by 160
  6. 2021Inductive Biases and Variable Creation in Self-Attention Mechanismscited by 144
  7. 2021Sinkformers: Transformers with Doubly Stochastic Attentioncited by 136
  8. 2022Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapsecited by 136
  9. 2023The emergence of clusters in self-attention dynamicscited by 133
  10. 2019Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of Viewcited by 108
  11. 2019Deep Learning via Dynamical Systems: An Approximation Perspectivecited by 78
  12. 2023Tighter Bounds on the Expressivity of Transformer Encoderscited by 65
35 more
  1. 2023Token Contrast for Weakly-Supervised Semantic Segmentationcited by 64
  2. 2021Neural ODE control for classification, approximation and transportcited by 56
  3. 2022Rank Diminishing in Deep Neural Networkscited by 47
  4. 2023Demystifying Oversmoothing in Attention-Based Graph Neural Networkscited by 47
  5. 2023The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limitcited by 47
  6. 2024On the Role of Attention Masks and LayerNorm in Transformerscited by 45
  7. 2020A Mathematical Theory of Attentioncited by 43
  8. 2024Transformers are Universal In-context Learnerscited by 42
  9. 2024Dynamic metastability in the self-attention modelcited by 42
  10. 2023ContraNorm: A Contrastive Learning Perspective on Oversmoothing and Beyondcited by 39
  11. 2022The Principle of Diversity: Training Stronger Vision Transformers Calls for Reducing All Levels of Redundancycited by 35
  12. 2024Emergence of meta-stable clustering in mean-field transformer modelscited by 35
  13. 2021Universal Approximation Under Constraints is Possible with Transformerscited by 32
  14. 2024Residual Connections and Normalization Can Provably Prevent Oversmoothing in GNNscited by 31
  15. 2023Sumformer: Universal Approximation for Efficient Transformerscited by 30
  16. 2024Understanding the Expressive Power and Mechanisms of Transformer for Sequence Modelingcited by 25
  17. 2024Prompting a Pretrained Transformer Can Be a Universal Approximatorcited by 25
  18. 2024Geometric Dynamics of Signal Propagation Predict Trainability of Transformerscited by 24
  19. 2024Provably learning a multi-head attention layercited by 21
  20. 2023Algorithms for mean-field variational inference via polyhedral optimization in the Wasserstein spacecited by 20
  21. 2024Setting the Record Straight on Transformer Oversmoothingcited by 19
  22. 2024Self-attention Networks Localize When QK-eigenspectrum Concentratescited by 19
  23. 2024Synchronization on circles and spheres with nonlinear interactionscited by 17
  24. 2023A Brief Survey on the Approximation Theory for Sequence Modellingcited by 15
  25. 2023Interpolation, Approximation and Controllability of Deep Neural Networkscited by 15
  26. 2024Clustering in pure-attention hardmax transformers and its role in sentiment analysiscited by 15
  27. 2015Optimal transport over a linear dynamical systemcited by 14
  28. 2023On the impact of activation and normalization in obtaining isometric embeddings at initializationcited by 13
  29. 2024Towards Understanding the Universality of Transformers for Next-Token Predictioncited by 9
  30. 2019Control in the spaces of ensembles of pointscited by 8
  31. 2022Neural ODE Control for Trajectory Approximation of Continuity Equationcited by 8
  32. 2024Generic controllability of equivariant systems and applications to particle systems and neural networkscited by 8
  33. 2021Deep Learning Approximation of Diffeomorphisms via Linear-Control Systemscited by 7
  34. 2017Approximate and exact controllability of the continuity equation with a localized vector fieldcited by 4
  35. 2024Some Remarks on Controllability of the Liouville Equationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 13; 2026: 13.

28 papers.

Sort:

  1. 2025A multiscale analysis of mean-field transformers in the moderate interaction regimecited by 24
  2. 2025The Intrinsic Dimension of Prompts in Internal Representations of Large Language Modelscited by 17
  3. 2025Learning Enhanced Ensemble Filterscited by 11
  4. 2025Normalization in Attention Dynamicscited by 9
  5. 2025Optimal Control for Transformer Architectures: Enhancing Generalization, Robustness and Efficiencycited by 8
  6. 2025OT-Transformer: A Continuous-time Transformer Architecture with Optimal Transport Regularizationcited by 7
  7. 2026Perceptrons and localization of attention's mean-field landscapecited by 7
  8. 2025Transformers through the lens of support-preserving maps between measurescited by 5
  9. 2025Optimal and Diffusion Transports in Machine Learningcited by 5
  10. 2024On the number of modes of Gaussian kernel density estimatorscited by 4
  11. 2024Constructive approximate transport maps with normalizing flowscited by 4
  12. 2025Exact Sequence Interpolation with Transformerscited by 2
16 more
  1. 2025When Do Transformers Outperform Feedforward and Recurrent Networks? A Statistical Perspectivecited by 2
  2. 2025A unified framework for establishing the universal approximation of transformer-type architecturescited by 2
  3. 2025Sliced ReLU attention: Quasi-linear contextual expressivity via sortingcited by 2
  4. 2026Propagation of Chaos in Contextual Flow Mapscited by 2
  5. 2026Function graph transformers universally approximate operators between function spacescited by 2
  6. 2026On the Expressive Power of Transformers for Contextual Relationscited by 1
  7. 2026Measure-to-measure Regression with Transformerscited by 1
  8. 2025Dual Filter: A Transformer-like Inference Architecture for Hidden Markov Modelscited by 0
  9. 2026Inverse Problems Conditioned on Observation Ensembles: Applications and Methodscited by 0
  10. 2026Kinetic theory for Transformers and the lost-in-the-middle phenomenoncited by 0
  11. 2026Transformer-like Inference from Optimal Controlcited by 0
  12. 2026Multi-Headed Transformer Architectures as Time-dependent Wasserstein Gradient Flowscited by 0
  13. 2026Reachability and asymptotics of Gaussian Transformer dynamicscited by 0
  14. 2026From Distributions to Stochastic Processes: Neural Approximation of Measure-Valued Mapscited by 0
  15. 2026Universal Approximation of Measure-to-Measure Operators by Pushforwardscited by 0
  16. 2026Universal interpolation for deep residual self-attention networkscited by 0