mlsanityData to 14 Oct 2026

LilNetX: Lightweight Networks with EXtreme Model Compression and Structured Sparsification

arXiv 2204.02965 ↗First posted 6 Apr 2022cs.CV cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

We introduce LilNetX, an end-to-end trainable technique for neural networks that enables learning models with specified accuracy-rate-computation trade-off. Prior works approach these problems one at a time and often require post-processing or multistage training which become less practical and do not scale very well for large datasets or architectures. Our method constructs a joint training objective that penalizes the self-information of network parameters in a reparameterized latent space to encourage small model size while also introducing priors to increase structured sparsity in the parameter space to reduce computation. We achieve up to 50% smaller model size and 98% model sparsity on ResNet-20 while retaining the same accuracy on the CIFAR-10 dataset as well as 35% smaller model size and 42% structured sparsity on ResNet-50 trained on ImageNet, when compared to existing state-of-the-art model compression methods. Code is available at https://github.com/Sharath-girish/LilNetX.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 75 of its 97 references that are in the map. First posted on arXiv 6 Apr 2022.

Builds on

75 of 97 references.

References · 75

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  5. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
  6. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  7. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  8. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  9. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  10. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  11. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  12. 2016Pruning Filters for Efficient ConvNetscited by 1,143
38 more
  1. 2018Variational image compression with a scale hyperpriorcited by 935
  2. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 849
  3. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 812
  4. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 702
  5. 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 676
  6. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 637
  7. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 592
  8. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 583
  9. 2016Learning Structured Sparsity in Deep Neural Networkscited by 541
  10. 2019Rigging the Lottery: Making All Tickets Winnerscited by 495
  11. 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 479
  12. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 471
  13. 2018HAQ: Hardware-Aware Automated Quantization with Mixed Precisioncited by 443
  14. 2017ThiNet: A Filter Level Pruning Method for Deep Neural Network Compressioncited by 442
  15. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 426
  16. 2018Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Accelerationcited by 367
  17. 2019Data-Free Quantization Through Weight Equalization and Bias Correctioncited by 336
  18. 2018Soft Filter Pruning for Accelerating Deep Convolutional Neural Networkscited by 315
  19. 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
  20. 2018LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networkscited by 274
  21. 2021NeRV: Neural Representations for Videoscited by 256
  22. 2020Comparing Rewinding and Fine-tuning in Neural Network Pruningcited by 240
  23. 2017Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weightscited by 217
  24. 2015Quantized Convolutional Neural Networks for Mobile Devicescited by 215
  25. 2016Trained Ternary Quantizationcited by 208
  26. 2016Ternary Weight Networkscited by 204
  27. 2020Proving the Lottery Ticket Hypothesis: Pruning is All You Needcited by 201
  28. 2014Compressing Deep Convolutional Networks using Vector Quantizationcited by 195
  29. 2020Pruning Neural Networks at Initialization: Why are We Missing the Mark?cited by 182
  30. 2017Data-Driven Sparse Structure Selection for Deep Neural Networkscited by 181
  31. 2021Learning N:M Fine-grained Structured Sparse Neural Networks From Scratchcited by 179
  32. 2018Discrimination-aware Channel Pruning for Deep Neural Networkscited by 171
  33. 2019Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networkscited by 160
  34. 2019Improving Neural Network Quantization without Retraining using Outlier Channel Splittingcited by 151
  35. 2015Compressing Neural Networks with the Hashing Trickcited by 150
  36. 2019Gate Decorator: Global Filter Pruning Method for Accelerating Deep Convolutional Neural Networkscited by 118
  37. 2017Bayesian Compression for Deep Learningcited by 116
  38. 2019Winning the Lottery with Continuous Sparsificationcited by 100

50 most cited of 75 All 75 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 4; 2024: 2.

7 papers.

  1. 2023EAGLES: Efficient Accelerated 3D Gaussians with Lightweight EncodingScited by 196
  2. 2022NIRVANA: Neural Implicit Representations of Videos with Adaptive Networks and Autoregressive Patch-wise Modelingcited by 51
  3. 2023SHACIRA: Scalable HAsh-grid Compression for Implicit Neural Representationscited by 31
  4. 2023ECRF: Entropy-Constrained Neural Radiance Fields Compression with Frequency Domain Optimizationcited by 8
  5. 2024Neural NeRF Compressioncited by 4
  6. 2024MCNC: Manifold-Constrained Reparameterization for Neural Compressioncited by 4
  7. 2023Bandwidth-efficient Inference for Neural Image Compressioncited by 1