# Neural Network Training Using ℓ1-Regularization and Bi-fidelity Data

arXiv: https://arxiv.org/abs/2105.13011 · HTML: https://mlsanity.com/paper/2105.13011/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 70 references that are in the map.
- First posted on arXiv 27 May 2021.

## Builds on (most cited first, 27 of 27)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [An overview of gradient descent optimization algorithms](https://mlsanity.com/paper/1609.04747/) (2016)
- [Reconciling modern machine learning practice and the bias-variance trade-off](https://mlsanity.com/paper/1812.11118/) (2018)
- [PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning](https://mlsanity.com/paper/1711.05769/) (2017)
- [Learning Efficient Convolutional Networks through Network Slimming](https://mlsanity.com/paper/1708.06519/) (2017)
- [Rethinking the Value of Network Pruning](https://mlsanity.com/paper/1810.05270/) (2018)
- [Error bounds for approximations with deep ReLU networks](https://mlsanity.com/paper/1610.01145/) (2016)
- [Learning Structured Sparsity in Deep Neural Networks](https://mlsanity.com/paper/1608.03665/) (2016)
- [Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks](https://mlsanity.com/paper/2102.00554/) (2021)
- [Adversarial Autoencoders](https://mlsanity.com/paper/1511.05644/) (2015)
- [A Survey of Model Compression and Acceleration for Deep Neural Networks](https://mlsanity.com/paper/1710.09282/) (2017)
- [Hidden Physics Models: Machine Learning of Nonlinear Partial Differential Equations](https://mlsanity.com/paper/1708.00588/) (2017)
- [Prediction of Aerodynamic Flow Fields Using Convolutional Neural Networks](https://mlsanity.com/paper/1905.13166/) (2019)
- [Group Sparse Regularization for Deep Neural Networks](https://mlsanity.com/paper/1607.00485/) (2016)
- [Fast ConvNets Using Group-wise Brain Damage](https://mlsanity.com/paper/1506.02515/) (2015)
- [Learning the Number of Neurons in Deep Networks](https://mlsanity.com/paper/1611.06321/) (2016)
- [DeepHoyer: Learning Sparser Neural Network with Differentiable Scale-Invariant Sparsity Measures](https://mlsanity.com/paper/1908.09979/) (2019)
- [The gap between theory and practice in function approximation with deep neural networks](https://mlsanity.com/paper/2001.07523/) (2020)
- [On transfer learning of neural networks using bi-fidelity data for uncertainty propagation](https://mlsanity.com/paper/2002.04495/) (2020)
- [Deep Neural Networks Are Effective At Learning High-Dimensional Hilbert-Valued Functions From Limited Data](https://mlsanity.com/paper/2012.06081/) (2020)
- [Memory Bounded Deep Convolutional Networks](https://mlsanity.com/paper/1412.1442/) (2014)
- [Topology Optimization under Uncertainty using a Stochastic Gradient-based Approach](https://mlsanity.com/paper/1902.04562/) (2019)
- [Bi-fidelity Stochastic Gradient Descent for Structural Optimization under Uncertainty](https://mlsanity.com/paper/1911.10420/) (2019)

## Built on by (most cited first, 16 of 16)

- [Multifidelity Deep Operator Networks For Data-Driven and Physics-Informed Problems](https://mlsanity.com/paper/2204.09157/) (2022)
- [Bi-fidelity Modeling of Uncertain and Partially Unknown Systems using DeepONets](https://mlsanity.com/paper/2204.00997/) (2022)
- [Multi-fidelity wavelet neural operator with application to uncertainty quantification](https://mlsanity.com/paper/2208.05606/) (2022)
- [Interpreting and generalizing deep learning in physics-based problems with functional linear models](https://mlsanity.com/paper/2307.04569/) (2023)
- [Multifidelity Kolmogorov-Arnold Networks](https://mlsanity.com/paper/2410.14764/) (2024)
- [Feature-adjacent multi-fidelity physics-informed machine learning for partial differential equations](https://mlsanity.com/paper/2303.11577/) (2023)
- [Bi-fidelity Variational Auto-encoder for Uncertainty Quantification](https://mlsanity.com/paper/2305.16530/) (2023)
- [PINN surrogate of Li-ion battery models for parameter inference. Part I: Implementation and multi-fidelity hierarchies for the single-particle model](https://mlsanity.com/paper/2312.17329/) (2023)
- [Data-Efficient Deep Operator Network for Unsteady Flow: A Multi-Fidelity Approach with Physics-Guided Subsampling](https://mlsanity.com/paper/2503.17941/) (2025)
- [Multifidelity linear regression for scientific machine learning from scarce data](https://mlsanity.com/paper/2403.08627/) (2024)
- [On the Interplay Between Sparsity and Training in Deep Reinforcement Learning](https://mlsanity.com/paper/2501.16729/) (2025)
- [Empowering Bayesian Neural Networks with Functional Priors through Anchored Ensembling for Mechanics Surrogate Modeling Applications](https://mlsanity.com/paper/2409.05234/) (2024)
- [Mathematical Formalism for Memory Compression in Selective State Space Models](https://mlsanity.com/paper/2410.03158/) (2024)
- [CoNNect: Connectivity-Based Regularization for Structural Pruning](https://mlsanity.com/paper/2502.00744/) (2025)
- [Transfer learning for predicting source terms of principal component transport in chemically reactive flow](https://mlsanity.com/paper/2312.00356/) (2023)
- [Stochastic Subspace Descent Accelerated via Bi-fidelity Line Search](https://mlsanity.com/paper/2505.00162/) (2025)

## Abstract (arXiv)

With the capability of accurately representing a functional relationship between the inputs of a physical system's model and output quantities of interest, neural networks have become popular for surrogate modeling in scientific applications. However, as these networks are over-parameterized, their training often requires a large amount of data. To prevent overfitting and improve generalization error, regularization based on, e.g., ℓ1- and ℓ2-norms of the parameters is applied. Similarly, multiple connections of the network may be pruned to increase sparsity in the network parameters. In this paper, we explore the effects of sparsity promoting ℓ1-regularization on training neural networks when only a small training dataset from a high-fidelity model is available. As opposed to standard ℓ1-regularization that is known to be inadequate, we consider two variants of ℓ1-regularization informed by the parameters of an identical network trained using data from lower-fidelity models of the problem at hand. These bi-fidelity strategies are generalizations of transfer learning of neural networks that uses the parameters learned from a large low-fidelity dataset to efficiently train networks for a small high-fidelity dataset. We also compare the bi-fidelity strategies with two ℓ1-regularization methods that only use the high-fidelity dataset. Three numerical examples for propagating uncertainty through physical systems are used to show that the proposed bi-fidelity ℓ1-regularization strategies produce errors that are one order of magnitude smaller than those of networks trained only using datasets from the high-fidelity models.