mlsanityData to 14 Oct 2026

AttNS: Attention-Inspired Numerical Solving For Limited Data Scenarios

arXiv 2302.10184 ↗First posted 5 Feb 2023cs.LG cs.AI cs.NAcited by 4

Not on the map. Why?

Abstract (arXiv)

We propose the attention-inspired numerical solver (AttNS), a concise method that helps the generalization and robustness issues faced by the AI-Hybrid numerical solver in solving differential equations due to limited data. AttNS is inspired by the effectiveness of attention modules in Residual Neural Networks (ResNet) in enhancing model generalization and robustness for conventional deep learning tasks. Drawing from the dynamical system perspective of ResNet, we seamlessly incorporate attention mechanisms into the design of numerical methods tailored for the characteristics of solving differential equations. Our results on benchmarks, ranging from high-dimensional problems to chaotic systems, showcases AttNS consistently enhancing various numerical solvers without any intricate model crafting. Finally, we analyze AttNS experimentally and theoretically, demonstrating its ability to achieve strong generalization and robustness while ensuring the convergence of the solver. This includes requiring less data compared to other advanced methods to achieve comparable generalization errors and better prevention of numerical explosion issues when solving differential equations.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 36 of its 70 references that are in the map. First posted on arXiv 5 Feb 2023.

Builds on

36 of 70 references.

References · 36

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  4. 2017Squeeze-and-Excitation Networkscited by 4,789
  5. 2018Neural Ordinary Differential Equationscited by 3,779
  6. 2020Fourier Neural Operator for Parametric Partial Differential Equationscited by 2,908
  7. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  8. 2018CBAM: Convolutional Block Attention Modulecited by 2,477
  9. 2019Hamiltonian Neural Networkscited by 712
  10. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  11. 2017Exploring Generalization in Deep Learningcited by 600
  12. 2019ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networkscited by 575
24 more
  1. 2021Physics-informed neural networks (PINNs) for fluid mechanics: A reviewcited by 473
  2. 2021Machine learning accelerated computational fluid dynamicscited by 471
  3. 2019Augmented Neural ODEscited by 397
  4. 2020Solver-in-the-Loop: Learning from Differentiable Physics to Interact with Iterative PDE-Solverscited by 205
  5. 2017Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equationscited by 203
  6. 2020FcaNet: Frequency Channel Attention Networkscited by 179
  7. 2020Transformers for Modeling Physical Systemscited by 106
  8. 2020Rational neural networkscited by 75
  9. 2021Fastformer: Additive Attention Can Be All You Needcited by 61
  10. 2021A Dynamical System Perspective for Lipschitz Neural Networkscited by 60
  11. 2017Multi-level Residual Networks from Dynamical Systems Viewcited by 59
  12. 2022Learning to correct spectral methods for simulating turbulent flowscited by 49
  13. 2020Continuous-in-Depth Neural Networkscited by 40
  14. 2022Learned Turbulence Modelling with Differentiable Fluid Solvers: Physics-based Loss-functions and Optimisation Horizonscited by 39
  15. 2019DIANet: Dense-and-Implicit Attention Networkcited by 37
  16. 2022A posteriori learning for quasi-geostrophic turbulence parametrizationcited by 36
  17. 2020"Hey, that's not an ODE": Faster ODE Adjoints via Seminormscited by 35
  18. 2019Instance Enhancement Batch Normalization: an Adaptive Regulator of Batch Noisecited by 28
  19. 2018Convolutional Neural Networks combined with Runge-Kutta Methodscited by 18
  20. 2018A machine learning framework for data driven acceleration of computations of differential equationscited by 18
  21. 2021Solving PDEs on Unknown Manifolds with Machine Learningcited by 12
  22. 2021Revisiting the Onsets and Frames Model with Additive Attentioncited by 9
  23. 2022On Fast Simulation of Dynamical System with Neural Vector Enhanced Numerical Solvercited by 9
  24. 2022Efficient Climate Simulation via Machine Learning Methodcited by 1

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 1; 2025: 2.

4 papers.

  1. 2025RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMscited by 61
  2. 2023ASR: Attention-alike Structural Re-parameterizationcited by 2
  3. 2024Learning Epidemiological Dynamics via the Finite Expression Methodcited by 2
  4. 2025Structurally Refined Graph Transformer for Multimodal Recommendationcited by 0