mlsanityData to 7 Oct 2026

Quadratic Residual Networks: A New Class of Neural Networks for Solving Forward and Inverse Problems in Physics Involving PDEs

arXiv 2101.08366 ↗First posted 20 Jan 2021cs.LG cs.CEcited by 42

Not on the map. Why?

Abstract (arXiv)

We propose quadratic residual networks (QRes) as a new type of parameter-efficient neural network architecture, by adding a quadratic residual term to the weighted sum of inputs before applying activation functions. With sufficiently high functional capacity (or expressive power), we show that it is especially powerful for solving forward and inverse physics problems involving partial differential equations (PDEs). Using tools from algebraic geometry, we theoretically demonstrate that, in contrast to plain neural networks, QRes shows better parameter efficiency in terms of network width and depth thanks to higher non-linearity in every neuron. Finally, we empirically show that QRes shows faster convergence speed in terms of number of training epochs especially in learning complex patterns.

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Builds on 18 of its 30 references that are in the map. First posted on arXiv 20 Jan 2021.

Builds on

18 of 30 references.

References · 18

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
  3. 2020When and why PINNs fail to train: A neural tangent kernel perspectivecited by 629
  4. 2017Physics Informed Deep Learning (Part I): Data-driven Solutions of Nonlinear Partial Differential Equationscited by 362
  5. 2019Universal Approximation with Deep Narrow Networkscited by 198
  6. 2016Theory-guided Data Science: A New Paradigm for Scientific Discovery from Datacited by 169
  7. 2017Physics Informed Deep Learning (Part II): Data-driven Discovery of Nonlinear Partial Differential Equationscited by 158
  8. 2019The Convergence Rate of Neural Networks for Learned Functions of Different Frequenciescited by 134
  9. 2018On the Power of Over-parametrization in Neural Networks with Quadratic Activationcited by 102
  10. 2018Neural Arithmetic Logic Unitscited by 69
  11. 2014A Stochastic Quasi-Newton Method for Large-Scale Optimizationcited by 68
  12. 2019On the Expressive Power of Deep Polynomial Neural Networkscited by 65
6 more
  1. 2018Universal Approximation with Quadratic Deep Networkscited by 29
  2. 2020Neural Arithmetic Unitscited by 25
  3. 2017A New Type of Neurons for Machine Learningcited by 16
  4. 2020A Survey on Activation Functions and their relation with Xavier and He Normal Initializationcited by 13
  5. 2019Exploring Self-Supervised Regularization for Supervised and Semi-Supervised Learningcited by 4
  6. 2015A Neural Transfer Function for a Smooth and Differentiable Transition Between Additive and Multiplicative Interactionscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 4; 2023: 9; 2024: 8; 2025: 11; 2026: 8.

42 papers.

Sort:

  1. 2022Respecting causality is all you need for training physics-informed neural networkscited by 154
  2. 2023PINNsFormer: A Transformer-Based Framework For Physics-Informed Neural Networkscited by 62
  3. 2022Deep Learning Methods for Partial Differential Equations and Related Parameter Identification Problemscited by 34
  4. 2024RoPINN: Region Optimized Physics-Informed Neural Networkscited by 26
  5. 2022QuadraLib: A Performant Quadratic Neural Network Library for Architecture Optimization and Design Explorationcited by 21
  6. 2021On Expressivity and Trainability of Quadratic Networkscited by 19
  7. 2025Identifiability of Deep Polynomial Neural Networkscited by 12
  8. 2023Monte Carlo Neural PDE Solver for Learning PDEs via Probabilistic Representationcited by 11
  9. 2025FP64 is All You Need: Rethinking Failure Modes in Physics-Informed Neural Networkscited by 11
  10. 2025Sub-Sequential Physics-Informed Learning with State Space Modelcited by 10
  11. 2024A Gaussian Process Framework for Solving Forward and Inverse Problems Involving Nonlinear Partial Differential Equationscited by 8
  12. 2023Towards NeuroAI: Introducing Neuronal Diversity into Artificial Neural Networkscited by 6
30 more
  1. 2023One Neuron Saved Is One Neuron Earned: On Parametric Efficiency of Quadratic Networkscited by 6
  2. 2025ProPINN: Demystifying Propagation Failures in Physics-Informed Neural Networkscited by 6
  3. 2025The Alexander-Hirschowitz theorem for neurovarietiescited by 6
  4. 2022Quadratic Neuron-empowered Heterogeneous Autoencoder for Unsupervised Anomaly Detectioncited by 5
  5. 2023Learning to Predict Structural Vibrationscited by 4
  6. 2021Learning Compact Representations of Neural Networks using DiscriminAtive Masking (DAM)cited by 3
  7. 2023A Sequential Meta-Transfer (SMT) Learning to Combat Complexities of Physics-Informed Neural Networks: Application to Composites Autoclave Processingcited by 3
  8. 2024HOIN: High-Order Implicit Neural Representationscited by 3
  9. 2024No One-Size-Fits-All Neurons: Task-based Neurons for Artificial Neural Networkscited by 3
  10. 2025Revisiting Feature Interactions from the Perspective of Quadratic Neural Networks for Click-through Rate Predictioncited by 3
  11. 2025Neuro-Spectral Architectures for Causal Physics-Informed Networkscited by 3
  12. 2024PAON: A New Neuron Model using Pad'e Approximantscited by 2
  13. 2024SetPINNs: Set-based Physics-informed Neural Networkscited by 2
  14. 2025Revisiting Heat Flux Analysis of Tungsten Monoblock Divertor on EAST using Physics-Informed Neural Networkcited by 2
  15. 2023Cloud-RAIN: Point Cloud Analysis with Reflectional Invariancecited by 1
  16. 2023Computational and Storage Efficient Quadratic Neurons for Deep Neural Networkscited by 1
  17. 2025PIANO: Physics Informed Autoregressive Networkcited by 1
  18. 2026naPINN: Noise-Adaptive Physics-Informed Neural Networks for Recovering Physics from Corrupted Measurementcited by 1
  19. 2023Dendritic Integration Based Quadratic Neural Networks Outperform Traditional Aritificial Onescited by 0
  20. 2024xMLP: Revolutionizing Private Inference with Exclusive Square Activationcited by 0
  21. 2024Retinal Vessel Segmentation via Neuron Programmingcited by 0
  22. 2025MixFunn: A Neural Network for Differential Equations with Improved Generalization and Interpretabilitycited by 0
  23. 2025NeuronSeek: On Stability and Expressivity of Task-driven Neuronscited by 0
  24. 2026Pad'e Neurons for Efficient Neural Modelscited by 0
  25. 2026A Family of Adaptive Activation Functions for Mitigating Failure Modes in Physics-Informed Neural Networkscited by 0
  26. 2026Multi-dimensional training-priority weighting based on physical information propagation paths: a unified residual-weighting framework for physics-informed neural networkscited by 0
  27. 2026Perturbative-NeuSA: A Structured Spectral Framework for Time-Dependent PDEscited by 0
  28. 2026Exact Algebraic Computation of Learning Coefficients for Two-Dimensional Singular Modelscited by 0
  29. 2026TNLearn: An Open Source Python Package for Task-based Neuronscited by 0
  30. 2026NEXT: Physics-Informed Neuro-Spectral Exponential Time Differencing Architecturescited by 0