# LightLayers: Parameter Efficient Dense and Convolutional Layers for Image Classification

arXiv: https://arxiv.org/abs/2101.02268 · HTML: https://mlsanity.com/paper/2101.02268/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 28 references that are in the map.
- First posted on arXiv 6 Jan 2021.

## Builds on (most cited first, 16 of 16)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation](https://mlsanity.com/paper/1808.00897/) (2018)
- [Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation](https://mlsanity.com/paper/1404.0736/) (2014)
- [ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation](https://mlsanity.com/paper/1606.02147/) (2016)
- [Tensorizing Neural Networks](https://mlsanity.com/paper/1509.06569/) (2015)
- [Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition](https://mlsanity.com/paper/1412.6553/) (2014)
- [Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applications](https://mlsanity.com/paper/1511.06530/) (2015)
- [Ultimate tensorization: compressing convolutional and FC layers alike](https://mlsanity.com/paper/1611.03214/) (2016)
- [CirCNN: Accelerating and Compressing Deep Neural Networks Using Block-CirculantWeight Matrices](https://mlsanity.com/paper/1708.08917/) (2017)
- [LEDNet: A Lightweight Encoder-Decoder Network for Real-Time Semantic Segmentation](https://mlsanity.com/paper/1905.02423/) (2019)
- [Diversity Networks: Neural Network Compression Using Determinantal Point Processes](https://mlsanity.com/paper/1511.05077/) (2015)
- [A Lightweight Deep Learning Model for Human Activity Recognition on Edge Devices](https://mlsanity.com/paper/1909.12917/) (2019)
- [An Extensive Study on Cross-Dataset Bias and Evaluation Metrics Interpretation for Machine Learning applied to Gastrointestinal Tract Abnormality Classification](https://mlsanity.com/paper/2005.03912/) (2020)
- [ProdSumNet: reducing model parameters in deep neural networks via product-of-sums matrix decompositions](https://mlsanity.com/paper/1809.02209/) (2018)
- [LRNNet: A Light-Weighted Network with Efficient Reduced Non-Local Operation for Real-Time Semantic Segmentation](https://mlsanity.com/paper/2006.02706/) (2020)

## Built on by (most cited first, 4 of 4)

- [WaveMix: A Resource-efficient Neural Network for Image Analysis](https://mlsanity.com/paper/2205.14375/) (2022)
- [Dynamic Neural Diversification: Path to Computationally Sustainable Neural Networks](https://mlsanity.com/paper/2109.09612/) (2021)
- [Unidirectional Thin Adapter for Efficient Adaptation of Deep Neural Networks](https://mlsanity.com/paper/2203.10463/) (2022)
- [DcnnGrasp: Towards Accurate Grasp Pattern Recognition with Adaptive Regularizer Learning](https://mlsanity.com/paper/2205.05218/) (2022)

## Abstract (arXiv)

Deep Neural Networks (DNNs) have become the de-facto standard in computer vision, as well as in many other pattern recognition tasks. A key drawback of DNNs is that the training phase can be very computationally expensive. Organizations or individuals that cannot afford purchasing state-of-the-art hardware or tapping into cloud-hosted infrastructures may face a long waiting time before the training completes or might not be able to train a model at all. Investigating novel ways to reduce the training time could be a potential solution to alleviate this drawback, and thus enabling more rapid development of new algorithms and models. In this paper, we propose LightLayers, a method for reducing the number of trainable parameters in deep neural networks (DNN). The proposed LightLayers consists of LightDense andLightConv2D layer that are as efficient as regular Conv2D and Dense layers, but uses less parameters. We resort to Matrix Factorization to reduce the complexity of the DNN models resulting into lightweight DNNmodels that require less computational power, without much loss in the accuracy. We have tested LightLayers on MNIST, Fashion MNIST, CI-FAR 10, and CIFAR 100 datasets. Promising results are obtained for MNIST, Fashion MNIST, CIFAR-10 datasets whereas CIFAR 100 shows acceptable performance by using fewer parameters.