mlsanityData to 14 Oct 2026

Forget the Data and Fine-Tuning! Just Fold the Network to Compress

arXiv 2502.10216 ↗First posted 14 Feb 2025cs.LG cs.AIcited by 7

Not on the map. Why?

Abstract (arXiv)

We introduce model folding, a novel data-free model compression technique that merges structurally similar neurons across layers, significantly reducing the model size without the need for fine-tuning or access to training data. Unlike existing methods, model folding preserves data statistics during compression by leveraging k-means clustering, and using novel data-free techniques to prevent variance collapse or explosion. Our theoretical framework and experiments across standard benchmarks, including ResNet18 and LLaMA-7B, demonstrate that model folding achieves comparable performance to data-driven compression techniques and outperforms recently proposed data-free methods, especially at high sparsity levels. This approach is particularly effective for compressing large-scale models, making it suitable for deployment in resource-constrained environments.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 54 of its 80 references that are in the map. First posted on arXiv 14 Feb 2025.

Builds on

54 of 80 references.

References · 54

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  6. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  8. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  9. 2016Pointer Sentinel Mixture Modelscited by 3,096
  10. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  11. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  12. 2020Knowledge Distillation: A Surveycited by 1,603
38 more
  1. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  2. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  3. 2016Pruning Filters for Efficient ConvNetscited by 1,143
  4. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 815
  5. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  6. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 787
  7. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 702
  8. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  9. 2021Merging Models with Fisher-Weighted Averagingcited by 621
  10. 2016Learning Structured Sparsity in Deep Neural Networkscited by 541
  11. 2017ThiNet: A Filter Level Pruning Method for Deep Neural Network Compressioncited by 442
  12. 2019Dreaming to Distill: Data-free Knowledge Transfer via DeepInversioncited by 423
  13. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 387
  14. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 385
  15. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 383
  16. 2015Deep Learning with Limited Numerical Precisioncited by 376
  17. 2017Implicit Regularization in Matrix Factorizationcited by 322
  18. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 322
  19. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 291
  20. 2016Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architecturescited by 221
  21. 2015Convergent Learning: Do different neural networks learn the same representations?cited by 220
  22. 2019Model Fusion via Optimal Transportcited by 219
  23. 2017Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weightscited by 217
  24. 2016Ternary Weight Networkscited by 204
  25. 2019Data-Free Learning of Student Networkscited by 204
  26. 2014Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decompositioncited by 203
  27. 2023ZipIt! Merging Models from Different Tasks without Trainingcited by 179
  28. 2015Compression of Deep Convolutional Neural Networks for Fast and Low Power Mobile Applicationscited by 162
  29. 2019Zero-shot Knowledge Transfer via Adversarial Belief Matchingcited by 129
  30. 2022REPAIR: REnormalizing Permuted Activations for Interpolation Repaircited by 120
  31. 2023Fluctuation-based Adaptive Structured Pruning for Large Language Modelscited by 111
  32. 2019Data-Free Adversarial Distillationcited by 90
  33. 2018Multi-Task Zipping via Layer-wise Neuron Sharingcited by 20
  34. 2023Going Beyond Neural Network Feature Similarity: The Network Feature Complexity and Its Interpretation Using Category Theorycited by 12
  35. 2015k-Means Clustering Is Matrix Factorizationcited by 11
  36. 2023Low-Rank Prune-And-Factorize for Language Model Compressioncited by 11
  37. 2023Maestro: Uncovering Low-Rank Structures via Trainable Decompositioncited by 8
  38. 2024Towards Meta-Pruning via Optimal Transportcited by 8

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 5.

7 papers.

  1. 2026Cut Less, Fold More: Model Compression through the Lens of Projection Geometrycited by 4
  2. 2025BALF: Budgeted Activation-Aware Low-Rank Factorization for Fine-Tuning-Free Model Compressioncited by 2
  3. 2026GRAIL: Post-hoc Compensation by Linear Reconstruction for Compressed Networkscited by 1
  4. 2026DLR: Zero-Inference-Cost Latent Residuals for Low-Rank Pre-Trainingcited by 1
  5. 2025Smooth Model Compression without Fine-Tuningcited by 0
  6. 2026Big2Small: A Unifying Neural Network Framework for Model Compressioncited by 0
  7. 2026On the Interaction Between Model Compression and Test-Time Adaptationcited by 0