On Information Geometry and Iterative Optimization in Model Compression: Operator Factorization
Not on the map. Why?
Abstract (arXiv)
The ever-increasing parameter counts of deep learning models necessitate effective compression techniques for deployment on resource-constrained devices. This paper explores the application of information geometry, the study of density-induced metrics on parameter spaces, to analyze existing methods within the space of model compression, primarily focusing on operator factorization. Adopting this perspective highlights the core challenge: defining an optimal low-compute submanifold (or subset) and projecting onto it. We argue that many successful model compression approaches can be understood as implicitly approximating information divergences for this projection. We highlight that when compressing a pre-trained model, using information divergences is paramount for achieving improved zero-shot accuracy, yet this may no longer be the case when the model is fine-tuned. In such scenarios, trainability of bottlenecked models turns out to be far more important for achieving high compression ratios with minimal performance degradation, necessitating adoption of iterative methods. In this context, we prove convergence of iterative singular value thresholding for training neural networks subject to a soft rank constraint. To further illustrate the utility of this perspective, we showcase how simple modifications to existing methods through softer rank reduction result in improved performance under fixed compression rates.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 61 of its 90 references that are in the map. First posted on arXiv 12 Jul 2025.
Builds on
61 of 90 references.
References · 61
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2015Distilling the Knowledge in a Neural Networkcited by 10,076
- 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
- 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
- 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
- 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
- 2021ImageNet-21K Pretraining for the Massescited by 607
- 2019Importance Estimation for Neural Network Pruningcited by 531
- 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 521
- 2018Quantizing deep convolutional networks for efficient inference: A whitepapercited by 507
- 2014New insights and perspectives on the natural gradient methodcited by 435
38 more
- 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 413
- 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 354
- 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 317
- 2014Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluationcited by 304
- 2018Model compression via distillation and quantizationcited by 287
- 2014Speeding up Convolutional Neural Networks with Low Rank Expansionscited by 283
- 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 280
- 2020On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselinescited by 254
- 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 231
- 2019Limitations of the Empirical Fisher Approximation for Natural Gradient Descentcited by 213
- 2014Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decompositioncited by 199
- 2022Language model compression with weighted low-rank factorizationcited by 192
- 2023ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Modelscited by 156
- 2020Dynamic Model Pruning with Feedbackcited by 125
- 2017Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracycited by 100
- 2014Guaranteed Matrix Completion via Non-convex Factorizationcited by 81
- 2024Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methodscited by 80
- 2020Stable Low-rank Tensor Decomposition for Compression of Convolutional Neural Networkcited by 74
- 2021Pufferfish: Communication-efficient Models At No Extra Costcited by 61
- 2021Initialization and Regularization of Factorized Neural Layerscited by 58
- 2024Compressing Large Language Models using Low Rank and Low Precision Decompositioncited by 58
- 2019Lightweight and Efficient End-to-End Speech Recognition Using Low-Rank Transformercited by 47
- 2022Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equationscited by 42
- 2024LQER: Low-Rank Quantization Error Reconstruction for LLMscited by 39
- 2018GroupReduce: Block-Wise Low-Rank Approximation for Neural Language Model Shrinkingcited by 32
- 2024Compute Better Spent: Replacing Dense Layers with Structured Matricescited by 24
- 2018Trained Rank Pruning for Efficient Deep Neural Networkscited by 23
- 2023Model Compression in Practice: Lessons Learned from Practitioners Creating On-device Machine Learning Experiencescited by 21
- 2022CAP: Correlation-Aware Pruning for Highly-Accurate Sparse Vision Modelscited by 20
- 2024OATS: Outlier-Aware Pruning Through Sparse and Low Rank Decompositioncited by 19
- 2020Neural Parameter Allocation Searchcited by 15
- 2022Numerical Optimizations for Weighted Low-rank Estimation on Language Modelcited by 14
- 2024Weakly Convex Regularisers for Inverse Problems: Convergence of Critical Points and Primal-Dual Optimisationcited by 12
- 2024Searching for Efficient Linear Layers over a Continuous Space of Structured Matricescited by 11
- 2014Forward - Backward Greedy Algorithms for Atomic Norm Regularizationcited by 9
- 2020Principal Component Networks: Parameter Reduction Early in Trainingcited by 9
- 2014An inertial forward-backward algorithm for the minimization of the sum of two nonconvex functionscited by 8
- 2022Deep Linear Networks for Matrix Completion -- An Infinite Depth Limitcited by 8
50 most cited of 61 All 61 (JSON)
Built on by
Citing papers per year: 2025: 1.
1 paper.