# Perturbation of Deep Autoencoder Weights for Model Compression and Classification of Tabular Data

arXiv: https://arxiv.org/abs/2205.08358 · HTML: https://mlsanity.com/paper/2205.08358/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 44 references that are in the map.
- First posted on arXiv 17 May 2022.

## Builds on (most cited first, 21 of 21)

- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Learning Efficient Convolutional Networks through Network Slimming](https://mlsanity.com/paper/1708.06519/) (2017)
- [Tabular Data: Deep Learning is Not All You Need](https://mlsanity.com/paper/2106.03253/) (2021)
- [Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Science](https://mlsanity.com/paper/1707.04780/) (2017)
- [SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training](https://mlsanity.com/paper/2106.01342/) (2021)
- [Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks](https://mlsanity.com/paper/1808.06866/) (2018)
- [Generating 3D faces using Convolutional Mesh Autoencoders](https://mlsanity.com/paper/1807.10267/) (2018)
- [Pruning and Quantization for Deep Neural Network Acceleration: A Survey](https://mlsanity.com/paper/2101.09671/) (2021)
- [Dynamic Network Surgery for Efficient DNNs](https://mlsanity.com/paper/1608.04493/) (2016)
- [Neural Oblivious Decision Ensembles for Deep Learning on Tabular Data](https://mlsanity.com/paper/1909.06312/) (2019)
- [What Do Compressed Deep Neural Networks Forget?](https://mlsanity.com/paper/1911.05248/) (2019)
- [Deep Rewiring: Training very sparse deep networks](https://mlsanity.com/paper/1711.05136/) (2017)
- [Dynamic Model Pruning with Feedback](https://mlsanity.com/paper/2006.07253/) (2020)
- [SubTab: Subsetting Features of Tabular Data for Self-Supervised Representation Learning](https://mlsanity.com/paper/2110.04361/) (2021)
- [Lookahead: A Far-Sighted Alternative of Magnitude-based Pruning](https://mlsanity.com/paper/2002.04809/) (2020)
- [Survey on Deep Neural Networks in Speech and Vision Systems](https://mlsanity.com/paper/1908.07656/) (2019)
- [AutoDropout: Learning Dropout Patterns to Regularize Deep Networks](https://mlsanity.com/paper/2101.01761/) (2021)
- [Towards Compact and Robust Deep Neural Networks](https://mlsanity.com/paper/1906.06110/) (2019)
- [EDropout: Energy-Based Dropout and Pruning of Deep Neural Networks](https://mlsanity.com/paper/2006.04270/) (2020)
- [Continuous Dropout](https://mlsanity.com/paper/1911.12675/) (2019)

## Built on by (most cited first, 7 of 7)

- [A Survey on Deep Tabular Learning](https://mlsanity.com/paper/2410.12034/) (2024)
- [Deep Clustering of Tabular Data by Weighted Gaussian Distribution Learning](https://mlsanity.com/paper/2301.00802/) (2023)
- [Attention versus Contrastive Learning of Tabular Data -- A Data-centric Benchmarking](https://mlsanity.com/paper/2401.04266/) (2024)
- [Effectiveness of Deep Image Embedding Clustering Methods on Tabular Data](https://mlsanity.com/paper/2212.14111/) (2022)
- [Between-Sample Relationship in Learning Tabular Data Using Graph and Attention Networks](https://mlsanity.com/paper/2306.06772/) (2023)
- [Predicting Early and Complete Drug Release from Long-Acting Injectables Using Explainable Machine Learning](https://mlsanity.com/paper/2601.02265/) (2026)
- [Generalized Context in Cross Attention for Transfer Learning of Disjoint Tabular Data](https://mlsanity.com/paper/2608.28209/) (2026)

## Abstract (arXiv)

Fully connected deep neural networks (DNN) often include redundant weights leading to overfitting and high memory requirements. Additionally, the performance of DNN is often challenged by traditional machine learning models in tabular data classification. In this paper, we propose periodical perturbations (prune and regrow) of DNN weights, especially at the self-supervised pre-training stage of deep autoencoders. The proposed weight perturbation strategy outperforms dropout learning in four out of six tabular data sets in downstream classification tasks. The L1 or L2 regularization of weights at the same pretraining stage results in inferior classification performance compared to dropout or our weight perturbation routine. Unlike dropout learning, the proposed weight perturbation routine additionally achieves 15% to 40% sparsity across six tabular data sets for the compression of deep pretrained models. Our experiments reveal that a pretrained deep autoencoder with weight perturbation or dropout can outperform traditional machine learning in tabular data classification when fully connected DNN fails miserably. However, traditional machine learning models appear superior to any deep models when a tabular data set contains uncorrelated variables. Therefore, the success of deep models can be attributed to the inevitable presence of correlated variables in real-world data sets.