mlsanityData to 14 Oct 2026

Compression-aware Training of Deep Networks

arXiv 1711.02638 ↗First posted 7 Nov 2017cs.CVcited by 46

Not on the map. Why?

Abstract (arXiv)

In recent years, great progress has been made in a variety of application domains thanks to the development of increasingly deeper neural networks. Unfortunately, the huge number of units of these networks makes them expensive both computationally and memory-wise. To overcome this, exploiting the fact that deep networks are over-parametrized, several compression strategies have been proposed. These methods, however, typically start from a network that has been trained in a standard manner, without considering such a future compression. In this paper, we propose to explicitly account for compression in the training process. To this end, we introduce a regularizer that encourages the parameter matrix of each layer to have low rank during training. We show that accounting for compression during training allows us to learn much more compact, yet at least as effective, models than state-of-the-art compression techniques.

Key facts. Cited by 46 papers in the map (data through 14 Oct 2026). First posted on arXiv 7 Nov 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 11; 2022: 13; 2023: 11; 2024: 5; 2025: 4; 2026: 2.

46 papers.

Sort:

By area:

  1. 2022Implicit Regularization Towards Rank Minimization in ReLU Networkscited by 53
  2. 2021Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decompositioncited by 33
  3. 2022Green Learning: Introduction, Examples and Outlookcited by 30
  4. 2023On the Embedding Collapse when Scaling up Recommendation Modelscited by 29
  5. 2021ChipNet: Budget-Aware Pruning with Heaviside Continuous Approximationscited by 21
  6. 2022Pruning Neural Networks via Coresets and Convex Geometry: Towards No Assumptionscited by 20
  7. 2022SGD and Weight Decay Secretly Minimize the Rank of Your Neural Networkcited by 19
  8. 2022Compression-aware Training of Neural Networks using Frank-Wolfecited by 16
  9. 2021Convolutional Neural Network Compression through Generalized Kronecker Product Decompositioncited by 14
  10. 2023A Survey on Approximate Edge AI for Energy Efficient Autonomous Driving Servicescited by 14
  11. 2024HarmoDT: Harmony Multi-Task Decision Transformer for Offline Reinforcement Learningcited by 14
  12. 2021Manipulating Identical Filter Redundancy for Efficient Pruning on Deep and Complicated CNNcited by 9
34 more
  1. 2022Boundary between noise and information applied to filtering neural network weight matricescited by 9
  2. 2023Efficient Neural Networks for Tiny Machine Learning: A Comprehensive Reviewcited by 9
  3. 2024Analysis of Linear Mode Connectivity via Permutation-Based Weight Matching: With Insights into Other Permutation Search Methodscited by 9
  4. 2021Distilling Image Classifiers in Object Detectorscited by 5
  5. 2022Pruning-aware Sparse Regularization for Network Pruningcited by 5
  6. 2023Low Rank Optimization for Efficient Deep Learning: Making A Balance between Compact Architecture and Fast Trainingcited by 5
  7. 2022Hyperspherical Quantization: Toward Smaller and More Accurate Modelscited by 4
  8. 2023Riemannian Low-Rank Model Compression for Federated Learning with Over-the-Air Aggregationcited by 4
  9. 2024The Fair Language Model Paradoxcited by 4
  10. 2021Model compression as constrained optimization, with application to neural nets. Part V: combining compressionscited by 3
  11. 2022Compact Model Training by Low-Rank Projection with Energy Transfercited by 3
  12. 2022Pushing the Efficiency Limit Using Structured Sparse Convolutionscited by 3
  13. 2023Federated Learning in IoT: a Survey from a Resource-Constrained Perspectivecited by 2
  14. 2024Towards Better Generalization: Weight Decay Induces Low-rank Bias for Neural Networkscited by 2
  15. 2021A Highly Effective Low-Rank Compression of Deep Neural Networks with Modified Beam-Search and Modified Stable Rankcited by 1
  16. 2022STN: Scalable Tensorizing Networks via Structure-Aware Training and Adaptive Compressioncited by 1
  17. 2023A Differentiable Framework for End-to-End Learning of Hybrid Structured Compressioncited by 1
  18. 2023Compression of Recurrent Neural Networks using Matrix Factorizationcited by 1
  19. 2024Task-Aware Harmony Multi-Task Decision Transformer for Offline Reinforcement Learningcited by 1
  20. 2025Compress to Impress: Efficient LLM Adaptation Using a Single Gradient Step on 100 Samplescited by 1
  21. 2025Q3R: Quadratic Reweighted Rank Regularizer for Effective Low-Rank Trainingcited by 1
  22. 2026CompSplat: Compression-aware 3D Gaussian Splatting for Real-world Videocited by 1
  23. 2026SLORR: Simple and Efficient In-Training Low-Rank Regularizationcited by 1
  24. 2021Compressing Deep CNNs using Basis Representation and Spectral Fine-tuningcited by 0
  25. 2021Pruning Ternary Quantizationcited by 0
  26. 2021Mining the Weights Knowledge for Optimizing Neural Network Structurescited by 0
  27. 2021An Acceleration Method Based on Deep Learning and Multilinear Feature Spacecited by 0
  28. 2022SeKron: A Decomposition Method Supporting Many Factorization Structurescited by 0
  29. 2022Pruning On-the-Fly: A Recoverable Pruning Method without Fine-tuningcited by 0
  30. 2023Progressive Ensemble Distillation: Building Ensembles for Efficient Inferencecited by 0
  31. 2023Survey on Computer Vision Techniques for Internet-of-Things Devicescited by 0
  32. 2023Enhancing Cross-Category Learning in Recommendation Systems with Multi-Layer Embedding Trainingcited by 0
  33. 2025You Don't Need All Attentions: Distributed Dynamic Fine-Tuning for Foundation Modelscited by 0
  34. 2025RocketStack: Level-aware Deep Recursive Ensemble Learning Architecturecited by 0