# PDE+: Enhancing Generalization via PDE with Adaptive Distributional Diffusion

arXiv: https://arxiv.org/abs/2305.15835 · HTML: https://mlsanity.com/paper/2305.15835/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 78 references that are in the map.
- First posted on arXiv 25 May 2023.

## Builds on (most cited first, 30 of 34)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Theoretically Principled Trade-off between Robustness and Accuracy](https://mlsanity.com/paper/1901.08573/) (2019)
- [Deeper, Broader and Artier Domain Generalization](https://mlsanity.com/paper/1710.03077/) (2017)
- [Robustness May Be at Odds with Accuracy](https://mlsanity.com/paper/1805.12152/) (2018)
- [Solving high-dimensional partial differential equations using deep learning](https://mlsanity.com/paper/1707.02568/) (2017)
- [Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness](https://mlsanity.com/paper/2006.10108/) (2020)
- [Stable Architectures for Deep Neural Networks](https://mlsanity.com/paper/1705.03341/) (2017)
- [Parseval Networks: Improving Robustness to Adversarial Examples](https://mlsanity.com/paper/1704.08847/) (2017)
- [Sorting out Lipschitz function approximation](https://mlsanity.com/paper/1811.05381/) (2018)
- [Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations](https://mlsanity.com/paper/1710.10121/) (2017)
- [Towards Robust Neural Networks via Random Self-ensemble](https://mlsanity.com/paper/1712.00673/) (2017)
- [Understanding and Mitigating the Tradeoff Between Robustness and Accuracy](https://mlsanity.com/paper/2002.10716/) (2020)
- [Robust Large Margin Deep Neural Networks](https://mlsanity.com/paper/1605.08254/) (2016)
- [PDE-GCN: Novel Architectures for Graph Neural Networks Motivated by Partial Differential Equations](https://mlsanity.com/paper/2108.01938/) (2021)
- [Embedding Propagation: Smoother Manifold for Few-Shot Classification](https://mlsanity.com/paper/2003.04151/) (2020)
- [On the effectiveness of adversarial training against common corruptions](https://mlsanity.com/paper/2103.02325/) (2021)
- [Noisy Activation Functions](https://mlsanity.com/paper/1603.00391/) (2016)
- [Limitations of the Lipschitz constant as a defense against adversarial examples](https://mlsanity.com/paper/1807.09705/) (2018)
- [A case for new neural network smoothness constraints](https://mlsanity.com/paper/2012.07969/) (2020)
- [Explicit Regularisation in Gaussian Noise Injections](https://mlsanity.com/paper/2007.07368/) (2020)
- [Noisy Feature Mixup](https://mlsanity.com/paper/2110.02180/) (2021)
- [Quantifying the generalization error in deep learning in terms of data distribution and neural network smoothness](https://mlsanity.com/paper/1905.11427/) (2019)

## Built on by (most cited first, 3 of 3)

- [TEA: Test-time Energy Adaptation](https://mlsanity.com/paper/2311.14402/) (2023)
- [Pretrained Reversible Generation as Unsupervised Visual Representation Learning](https://mlsanity.com/paper/2412.01787/) (2024)
- [MITA: Bridging the Gap between Model and Data for Test-time Adaptation](https://mlsanity.com/paper/2410.09398/) (2024)

## Abstract (arXiv)

The generalization of neural networks is a central challenge in machine learning, especially concerning the performance under distributions that differ from training ones. Current methods, mainly based on the data-driven paradigm such as data augmentation, adversarial training, and noise injection, may encounter limited generalization due to model non-smoothness. In this paper, we propose to investigate generalization from a Partial Differential Equation (PDE) perspective, aiming to enhance it directly through the underlying function of neural networks, rather than focusing on adjusting input data. Specifically, we first establish the connection between neural network generalization and the smoothness of the solution to a specific PDE, namely "transport equation". Building upon this, we propose a general framework that introduces adaptive distributional diffusion into transport equation to enhance the smoothness of its solution, thereby improving generalization. In the context of neural networks, we put this theoretical framework into practice as PDE+ (PDE with Adaptive Distributional Diffusion) which diffuses each sample into a distribution covering semantically similar inputs. This enables better coverage of potentially unobserved distributions in training, thus improving generalization beyond merely data-driven methods. The effectiveness of PDE+ is validated through extensive experimental settings, demonstrating its superior performance compared to SOTA methods.