# AttNS: Attention-Inspired Numerical Solving For Limited Data Scenarios

arXiv: https://arxiv.org/abs/2302.10184 · HTML: https://mlsanity.com/paper/2302.10184/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 70 references that are in the map.
- First posted on arXiv 5 Feb 2023.

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [Neural Tangent Kernel: Convergence and Generalization in Neural Networks](https://mlsanity.com/paper/1806.07572/) (2018)
- [CBAM: Convolutional Block Attention Module](https://mlsanity.com/paper/1807.06521/) (2018)
- [Hamiltonian Neural Networks](https://mlsanity.com/paper/1906.01563/) (2019)
- [Gradient Descent Provably Optimizes Over-parameterized Neural Networks](https://mlsanity.com/paper/1810.02054/) (2018)
- [Exploring Generalization in Deep Learning](https://mlsanity.com/paper/1706.08947/) (2017)
- [ECA-Net: Efficient Channel Attention for Deep Convolutional Neural Networks](https://mlsanity.com/paper/1910.03151/) (2019)
- [Physics-informed neural networks (PINNs) for fluid mechanics: A review](https://mlsanity.com/paper/2105.09506/) (2021)
- [Machine learning accelerated computational fluid dynamics](https://mlsanity.com/paper/2102.01010/) (2021)
- [Augmented Neural ODEs](https://mlsanity.com/paper/1904.01681/) (2019)
- [Solver-in-the-Loop: Learning from Differentiable Physics to Interact with Iterative PDE-Solvers](https://mlsanity.com/paper/2007.00016/) (2020)
- [Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations](https://mlsanity.com/paper/1710.10121/) (2017)
- [FcaNet: Frequency Channel Attention Networks](https://mlsanity.com/paper/2012.11879/) (2020)
- [Transformers for Modeling Physical Systems](https://mlsanity.com/paper/2010.03957/) (2020)
- [Rational neural networks](https://mlsanity.com/paper/2004.01902/) (2020)
- [Fastformer: Additive Attention Can Be All You Need](https://mlsanity.com/paper/2108.09084/) (2021)
- [A Dynamical System Perspective for Lipschitz Neural Networks](https://mlsanity.com/paper/2110.12690/) (2021)
- [Multi-level Residual Networks from Dynamical Systems View](https://mlsanity.com/paper/1710.10348/) (2017)
- [Learning to correct spectral methods for simulating turbulent flows](https://mlsanity.com/paper/2207.00556/) (2022)
- [Continuous-in-Depth Neural Networks](https://mlsanity.com/paper/2008.02389/) (2020)
- [Learned Turbulence Modelling with Differentiable Fluid Solvers: Physics-based Loss-functions and Optimisation Horizons](https://mlsanity.com/paper/2202.06988/) (2022)
- [DIANet: Dense-and-Implicit Attention Network](https://mlsanity.com/paper/1905.10671/) (2019)
- [A posteriori learning for quasi-geostrophic turbulence parametrization](https://mlsanity.com/paper/2204.03911/) (2022)
- ["Hey, that's not an ODE": Faster ODE Adjoints via Seminorms](https://mlsanity.com/paper/2009.09457/) (2020)
- [Instance Enhancement Batch Normalization: an Adaptive Regulator of Batch Noise](https://mlsanity.com/paper/1908.04008/) (2019)

## Built on by (most cited first, 4 of 4)

- [RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMs](https://mlsanity.com/paper/2503.10657/) (2025)
- [ASR: Attention-alike Structural Re-parameterization](https://mlsanity.com/paper/2304.06345/) (2023)
- [Learning Epidemiological Dynamics via the Finite Expression Method](https://mlsanity.com/paper/2412.21049/) (2024)
- [Structurally Refined Graph Transformer for Multimodal Recommendation](https://mlsanity.com/paper/2511.00584/) (2025)

## Abstract (arXiv)

We propose the attention-inspired numerical solver (AttNS), a concise method that helps the generalization and robustness issues faced by the AI-Hybrid numerical solver in solving differential equations due to limited data. AttNS is inspired by the effectiveness of attention modules in Residual Neural Networks (ResNet) in enhancing model generalization and robustness for conventional deep learning tasks. Drawing from the dynamical system perspective of ResNet, we seamlessly incorporate attention mechanisms into the design of numerical methods tailored for the characteristics of solving differential equations. Our results on benchmarks, ranging from high-dimensional problems to chaotic systems, showcases AttNS consistently enhancing various numerical solvers without any intricate model crafting. Finally, we analyze AttNS experimentally and theoretically, demonstrating its ability to achieve strong generalization and robustness while ensuring the convergence of the solver. This includes requiring less data compared to other advanced methods to achieve comparable generalization errors and better prevention of numerical explosion issues when solving differential equations.