# Optimizing Dense Feed-Forward Neural Networks

arXiv: https://arxiv.org/abs/2312.10560 · HTML: https://mlsanity.com/paper/2312.10560/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 31 references that are in the map.
- First posted on arXiv 16 Dec 2023.

## Builds on (most cited first, 13 of 13)

- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [Regularized Evolution for Image Classifier Architecture Search](https://mlsanity.com/paper/1802.01548/) (2018)
- [What is the State of Neural Network Pruning?](https://mlsanity.com/paper/2003.03033/) (2020)
- [Large-Scale Evolution of Image Classifiers](https://mlsanity.com/paper/1703.01041/) (2017)
- [Pruning and Quantization for Deep Neural Network Acceleration: A Survey](https://mlsanity.com/paper/2101.09671/) (2021)
- [Dynamic Network Surgery for Efficient DNNs](https://mlsanity.com/paper/1608.04493/) (2016)
- [Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon](https://mlsanity.com/paper/1705.07565/) (2017)
- [A Survey on Evolutionary Neural Architecture Search](https://mlsanity.com/paper/2008.10937/) (2020)
- [Hardware and Software Optimizations for Accelerating Deep Neural Networks: Survey of Current Trends, Challenges, and the Road Ahead](https://mlsanity.com/paper/2012.11233/) (2020)
- [Sub-Architecture Ensemble Pruning in Neural Architecture Search](https://mlsanity.com/paper/1910.00370/) (2019)
- [Automated Pruning for Deep Neural Network Compression](https://mlsanity.com/paper/1712.01721/) (2017)

## Built on by (most cited first, 2 of 2)

- [Simple Projection Variants Improve ColBERT Performance](https://mlsanity.com/paper/2510.12327/) (2025)
- [Optimizing Convolutional Neural Network Architecture](https://mlsanity.com/paper/2401.01361/) (2023)

## Abstract (arXiv)

Deep learning models have been widely used during the last decade due to their outstanding learning and abstraction capacities. However, one of the main challenges any scientist has to face using deep learning models is to establish the network's architecture. Due to this difficulty, data scientists usually build over complex models and, as a result, most of them result computationally intensive and impose a large memory footprint, generating huge costs, contributing to climate change and hindering their use in computational-limited devices. In this paper, we propose a novel feed-forward neural network constructing method based on pruning and transfer learning. Its performance has been thoroughly assessed in classification and regression problems. Without any accuracy loss, our approach can compress the number of parameters by more than 70%. Even further, choosing the pruning parameter carefully, most of the refined models outperform original ones. We also evaluate the transfer learning level comparing the refined model and the original one training from scratch a neural network with the same hyper parameters as the optimized model. The results obtained show that our constructing method not only helps in the design of more efficient models but also more effective ones.