# Sparse deep neural networks for modeling aluminum electrolysis dynamics

arXiv: https://arxiv.org/abs/2209.05832 · HTML: https://mlsanity.com/paper/2209.05832/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 49 references that are in the map.
- First posted on arXiv 13 Sep 2022.

## Builds on (most cited first, 13 of 13)

- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [To prune, or not to prune: exploring the efficacy of pruning for model compression](https://mlsanity.com/paper/1710.01878/) (2017)
- [Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks](https://mlsanity.com/paper/2102.00554/) (2021)
- [On the Number of Linear Regions of Deep Neural Networks](https://mlsanity.com/paper/1402.1869/) (2014)
- [A Survey on Neural Network Interpretability](https://mlsanity.com/paper/2012.14261/) (2020)
- [Bounding and Counting Linear Regions of Deep Neural Networks](https://mlsanity.com/paper/1711.02114/) (2017)
- [On the number of response regions of deep feed forward networks with piece-wise linear activations](https://mlsanity.com/paper/1312.6098/) (2013)
- [Physics guided machine learning using simplified theories](https://mlsanity.com/paper/2012.13343/) (2020)
- [Deep neural network enabled corrective source term approach to hybrid analysis and modeling](https://mlsanity.com/paper/2105.11521/) (2021)
- [Hybrid analysis and modeling, eclecticism, and multifidelity computing toward digital twin revolution](https://mlsanity.com/paper/2103.14629/) (2021)
- [Sparse Bayesian Deep Learning for Dynamic System Identification](https://mlsanity.com/paper/2107.12910/) (2021)
- [On improving deep learning generalization with adaptive sparse connectivity](https://mlsanity.com/paper/1906.11626/) (2019)
- [Combining physics-based and data-driven techniques for reliable hybrid analysis and modeling using the corrective source term approach](https://mlsanity.com/paper/2206.03451/) (2022)

## Built on by (most cited first, 3 of 3)

- [A novel corrective-source term approach to modeling unknown physics in aluminum extraction process](https://mlsanity.com/paper/2209.10861/) (2022)
- [Sparse neural networks with skip-connections for identification of aluminum electrolysis cell](https://mlsanity.com/paper/2301.00582/) (2023)
- [Deep active learning for nonlinear system identification](https://mlsanity.com/paper/2302.12667/) (2023)

## Abstract (arXiv)

Deep neural networks have become very popular in modeling complex nonlinear processes due to their extraordinary ability to fit arbitrary nonlinear functions from data with minimal expert intervention. However, they are almost always overparameterized and challenging to interpret due to their internal complexity. Furthermore, the optimization process to find the learned model parameters can be unstable due to the process getting stuck in local minima. In this work, we demonstrate the value of sparse regularization techniques to significantly reduce the model complexity. We demonstrate this for the case of an aluminium extraction process, which is highly nonlinear system with many interrelated subprocesses. We trained a densely connected deep neural network to model the process and then compared the effects of sparsity promoting l1 regularization on generalizability, interpretability, and training stability. We found that the regularization significantly reduces model complexity compared to a corresponding dense neural network. We argue that this makes the model more interpretable, and show that training an ensemble of sparse neural networks with different parameter initializations often converges to similar model structures with similar learned input features. Furthermore, the empirical study shows that the resulting sparse models generalize better from small training sets than their dense counterparts.