# S2GPT-PINNs: Sparse and Small models for PDEs

arXiv: https://arxiv.org/abs/2506.15687 · HTML: https://mlsanity.com/paper/2506.15687/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 35 references that are in the map.
- First posted on arXiv 25 May 2025.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://mlsanity.com/paper/2009.07118/) (2020)
- [Solving high-dimensional partial differential equations using deep learning](https://mlsanity.com/paper/1707.02568/) (2017)
- [Small Language Models: Survey, Measurements, and Insights](https://mlsanity.com/paper/2409.15790/) (2024)
- [A comprehensive deep learning-based approach to reduced order modeling of nonlinear time-dependent parametrized PDEs](https://mlsanity.com/paper/2001.04001/) (2020)
- [POD-DL-ROM: enhancing deep learning-based reduced order models for nonlinear parametrized PDEs by proper orthogonal decomposition](https://mlsanity.com/paper/2101.11845/) (2021)
- [Failure-informed adaptive sampling for PINNs](https://mlsanity.com/paper/2210.00279/) (2022)
- [Meta-Auto-Decoder for Solving Parametric Partial Differential Equations](https://mlsanity.com/paper/2111.08823/) (2021)
- [PINNACLE: PINN Adaptive ColLocation and Experimental points selection](https://mlsanity.com/paper/2404.07662/) (2024)
- [Blending Is All You Need: Cheaper, Better Alternative to Trillion-Parameters LLM](https://mlsanity.com/paper/2401.02994/) (2024)
- [Galerkin Neural Networks: A Framework for Approximating Variational Equations with Error Control](https://mlsanity.com/paper/2105.14094/) (2021)
- [GPT-PINN: Generative Pre-Trained Physics-Informed Neural Networks toward non-intrusive Meta-learning of parametric PDEs](https://mlsanity.com/paper/2303.14878/) (2023)
- [TGPT-PINN: Nonlinear model reduction with transformed GPT-PINNs](https://mlsanity.com/paper/2403.03459/) (2024)

## Built on by (most cited first, 2 of 2)

- [Position: Certifiable State Integrity Should Be Built from Local Validity, Not Global Scale](https://mlsanity.com/paper/2601.21249/) (2026)
- [White-Box Neural Ensemble for Vehicular Plasticity: Quantifying the Efficiency Cost of Symbolic Auditability in Adaptive NMPC](https://mlsanity.com/paper/2602.01516/) (2026)

## Abstract (arXiv)

We propose S2GPT-PINN, a sparse and small model for solving parametric partial differential equations (PDEs). Similar to Small Language Models (SLMs), S2GPT-PINN is tailored to domain-specific (families of) PDEs and characterized by its compact architecture and minimal computational power. Leveraging a small amount of extremely high quality data via a mathematically rigorous greedy algorithm that is enabled by the large full-order models, S2GPT-PINN relies on orders of magnitude less parameters than PINNs to achieve extremely high efficiency via two levels of customizations. The first is knowledge distillation via task-specific activation functions that are transferred from Pre-Trained PINNs. The second is a judicious down-sampling when calculating the physics-informed loss of the network compressing the number of data sites by orders of magnitude to the size of the small model.