mlsanityData to 14 Oct 2026

Cut Less, Fold More: Model Compression through the Lens of Projection Geometry

arXiv 2602.18116 ↗First posted 20 Feb 2026cs.LG cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

Compressing neural networks without retraining is vital for deployment at scale. We study calibration-free compression through the lens of projection geometry: structured pruning is an axis-aligned projection, whereas model folding performs a low-rank projection via weight clustering. We formalize both as orthogonal operators and show that, within a rank distance of one, folding provably yields smaller parameter reconstruction error, and under mild smoothness assumptions, smaller functional perturbations than pruning. At scale, we evaluate >1000 checkpoints spanning ResNet18, PreActResNet18, ViT-B/32, and CLIP ViT-B/32 on CIFAR-10 and ImageNet-1K, covering diverse training hyperparameters (optimizers, learning rates, augmentations, regularization, sharpness-aware training), as well as multiple LLaMA-family 60M and 130M parameter models trained on C4. We show that folding typically achieves higher post-compression accuracy, with the largest gains at moderate-high compression. The gap narrows and occasionally reverses at specific training setups. Our results position folding as a geometry-aware, calibration-free alternative to pruning that is often superior in practice and principled in theory.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 52 of its 59 references that are in the map. First posted on arXiv 20 Feb 2026.

Builds on

52 of 59 references.

References · 52

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  6. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  7. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  8. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  9. 2016Pruning Filters for Efficient ConvNetscited by 1,143
  10. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
  11. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  12. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
38 more
  1. 2017Exploring Generalization in Deep Learningcited by 600
  2. 2016Learning Structured Sparsity in Deep Neural Networkscited by 541
  3. 2017ThiNet: A Filter Level Pruning Method for Deep Neural Network Compressioncited by 442
  4. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 403
  5. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 291
  6. 2017Three Factors Influencing Minima in SGDcited by 273
  7. 2017The Marginal Value of Adaptive Gradient Methods in Machine Learningcited by 250
  8. 2021Accelerating Sparse Deep Neural Networkscited by 223
  9. 2016Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architecturescited by 221
  10. 2019Data-Free Learning of Student Networkscited by 204
  11. 2014Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decompositioncited by 203
  12. 2023ZipIt! Merging Models from Different Tasks without Trainingcited by 179
  13. 2015Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applicationscited by 162
  14. 2019Zero-shot Knowledge Transfer via Adversarial Belief Matchingcited by 129
  15. 2020Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learningcited by 126
  16. 2022REPAIR: REnormalizing Permuted Activations for Interpolation Repaircited by 120
  17. 2022The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Modelscited by 115
  18. 2023A Modern Look at the Relationship between Sharpness and Generalizationcited by 92
  19. 2019Data-Free Adversarial Distillationcited by 90
  20. 2019Large Scale Structure of Neural Network Loss Landscapescited by 68
  21. 2019The Knowledge Within: Methods for Data-Free Model Compressioncited by 54
  22. 2024SLTrain: a sparse plus low-rank approach for parameter and memory efficient pretrainingcited by 45
  23. 2021FAST: DNN Training Under Variable Precision Block Floating Point with Stochastic Roundingcited by 31
  24. 2018Balanced Sparsity for Efficient DNN Inference on GPUcited by 27
  25. 2023STEP: Learning N:M Structured Sparsity Masks from Scratch with Preconditioncited by 26
  26. 2018Accelerator-Aware Pruning for Convolutional Neural Networkscited by 22
  27. 2024Progressive Gradient Flow for Robust N:M Sparsity Training in Transformerscited by 18
  28. 2023Adaptive Sharpness-Aware Pruning for Robust Sparse Networkscited by 17
  29. 2019Understanding Adversarial Robustness Through Loss Landscape Geometriescited by 12
  30. 2023Going Beyond Neural Network Feature Similarity: The Network Feature Complexity and Its Interpretation Using Category Theorycited by 12
  31. 2015k-Means Clustering Is Matrix Factorizationcited by 11
  32. 2023Low-Rank Prune-And-Factorize for Language Model Compressioncited by 11
  33. 2023How Sparse Can We Prune A Deep Network: A Fundamental Limit Perspectivecited by 9
  34. 2023Maestro: Uncovering Low-Rank Structures via Trainable Decompositioncited by 8
  35. 2024Towards Meta-Pruning via Optimal Transportcited by 8
  36. 2019Class-dependent Compression of Deep Neural Networkscited by 7
  37. 2025Forget the Data and Fine-Tuning! Just Fold the Network to Compresscited by 7
  38. 2025Signal Collapse in One-Shot Pruning: When Sparse Models Fail to Distinguish Neural Representationscited by 5

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Understanding Calibration and Truncation Error Propagation in Training-Free Low-Rank Compression for LLMscited by 2
  2. 2026GRAIL: Post-hoc Compensation by Linear Reconstruction for Compressed Networkscited by 1
  3. 2026DLR: Zero-Inference-Cost Latent Residuals for Low-Rank Pre-Trainingcited by 1
  4. 2026On the Interaction Between Model Compression and Test-Time Adaptationcited by 0