mlsanityData to 7 Oct 2026

A Brief Survey on the Approximation Theory for Sequence Modelling

arXiv 2302.13752 ↗First posted 27 Feb 2023cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

We survey current developments in the approximation theory of sequence modelling in machine learning. Particular emphasis is placed on classifying existing results for various model architectures through the lens of classical approximation paradigms, and the insights one can gain from these results. We also outline some future research directions towards building a theory of sequence modelling.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 49 of its 88 references that are in the map. First posted on arXiv 27 Feb 2023.

Builds on

49 of 88 references.

References · 49

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  4. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  5. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  6. 2019DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operatorscited by 1,965
  7. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  8. 2018An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modelingcited by 1,447
  9. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,303
  10. 2016Temporal Convolutional Networks for Action Segmentation and Detectioncited by 504
  11. 2020Neural Controlled Differential Equations for Irregular Time Seriescited by 501
  12. 2021Neural Operator: Learning Maps Between Function Spacescited by 413
37 more
  1. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 411
  2. 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 404
  3. 2019On the Relationship between Self-Attention and Convolutional Layerscited by 299
  4. 2018Universality of Deep Convolutional Neural Networkscited by 176
  5. 2021Neural SDEs as Infinite-Dimensional GANscited by 159
  6. 2020Neural Rough Differential Equations for Long Time Seriescited by 140
  7. 2018Universal approximations of invariant maps by neural networkscited by 127
  8. 2020Towards a Mathematical Understanding of Neural Network-Based Machine Learning: what we know and what we don'tcited by 94
  9. 2018Echo state networks are universalcited by 86
  10. 2018On the Convergence Rate of Training Recurrent Neural Networkscited by 82
  11. 2020O(n) Connections are Expressive Enough: Universal Approximability of Sparse Transformerscited by 82
  12. 2016Gradient Descent Learns Linear Dynamical Systemscited by 78
  13. 2018A Priori Estimates of the Population Risk for Two-layer Neural Networkscited by 76
  14. 2020Approximation Bounds for Random Neural Networks and Reservoir Systemscited by 52
  15. 2018Reservoir Computing Universality With Stochastic Inputscited by 48
  16. 2021Neural Stochastic PDEs: Resolution-Invariant Learning of Continuous Spatiotemporal Dynamicscited by 46
  17. 2019On Generalization Bounds of a Family of Recurrent Neural Networkscited by 42
  18. 2019Approximation and Non-parametric Estimation of ResNet-type Convolutional Neural Networkscited by 40
  19. 2017Universal discrete-time reservoir computers with stochastic inputs and linear readouts using non-homogeneous state-affine systemscited by 39
  20. 2020On the Banach spaces associated with multi-layer ReLU networks: Function representation, approximation theory and gradient descent dynamicscited by 39
  21. 2018Equivalence of approximation by convolutional neural networks and fully-connected networkscited by 38
  22. 2021Universal Approximation Under Constraints is Possible with Transformerscited by 32
  23. 2020Fading memory echo state networks are universalcited by 31
  24. 2020On the Curse of Memory in Recurrent Neural Networks: Approximation and Optimization Analysiscited by 26
  25. 2022Designing Universal Causal Deep Learning Models: The Geometric (Hyper)Transformercited by 22
  26. 2022Designing Universal Causal Deep Learning Models: The Case of Infinite-Dimensional Dynamical Systems from Stochastic Analysiscited by 22
  27. 2019Differentiable reservoir computingcited by 20
  28. 2021Can Vision Transformers Perform Convolution?cited by 18
  29. 2022Small Transformers Compute Universal Metric Embeddingscited by 12
  30. 2022Minimal Width for Universal Property of Deep RNNcited by 12
  31. 2022Neural Operator with Regularity Structure for Modeling Dynamics Driven by SPDEscited by 11
  32. 2021Approximation Theory of Convolutional Architectures for Time Series Modellingcited by 10
  33. 2022Chaotic Hedging with Iterated Integrals and Neural Networkscited by 10
  34. 2022Deep Neural Network Approximation of Invariant Functions through Dynamical Systemscited by 8
  35. 2021On the Provable Generalization of Recurrent Neural Networkscited by 6
  36. 2022On the Universal Approximation Property of Deep Fully Convolutional Neural Networkscited by 6
  37. 2023Neural SDEs for Conditional Time Series Generation and the Signature-Wasserstein-1 metriccited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 6; 2025: 1; 2026: 1.

15 papers.

Sort:

By area:

  1. 2023A mathematical perspective on Transformerscited by 160
  2. 2023State-space Models with Layer-wise Nonlinearity are Universal Approximators with Exponential Decaying Memorycited by 38
  3. 2023StableSSM: Alleviating the Curse of Memory in State-space Models through Stable Reparameterizationcited by 29
  4. 2024Measure-to-measure interpolation using Transformerscited by 28
  5. 2024Prompting a Pretrained Transformer Can Be a Universal Approximatorcited by 25
  6. 2023Inverse Approximation Theory for Nonlinear Recurrent Neural Networkscited by 10
  7. 2024Rethinking Transformers in Solving POMDPscited by 9
  8. 2023Parrot Mind: Towards Explaining the Complex Task Reasoning of Pretrained Large Language Models with Template-Content Structurecited by 3
  9. 2024State-space systems as dynamic generative modelscited by 3
  10. 2024LongSSM: On the Length Extension of State-space Models in Language Modellingcited by 3
  11. 2025Stochastic dynamics learning with state-space systemscited by 3
  12. 2024Fading memory and the convolution theoremcited by 2
3 more
  1. 2023Forward and Inverse Approximation Theory for Linear Temporal Convolutional Networkscited by 1
  2. 2023Improve Long-term Memory Learning Through Rescaling the Error Temporallycited by 1
  3. 2026Measure-to-measure Regression with Transformerscited by 1