# Big2Small: A Unifying Neural Network Framework for Model Compression

arXiv: https://arxiv.org/abs/2603.29768 · HTML: https://mlsanity.com/paper/2603.29768/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 53 of its 75 references that are in the map.
- First posted on arXiv 31 Mar 2026.

## Builds on (most cited first, 30 of 53)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [UNet++: A Nested U-Net Architecture for Medical Image Segmentation](https://mlsanity.com/paper/1807.10165/) (2018)
- [Swin Transformer V2: Scaling Up Capacity and Resolution](https://mlsanity.com/paper/2111.09883/) (2021)
- [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://mlsanity.com/paper/2301.00774/) (2023)
- [XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks](https://mlsanity.com/paper/1603.05279/) (2016)
- [Dreaming to Distill: Data-free Knowledge Transfer via DeepInversion](https://mlsanity.com/paper/1912.08795/) (2019)
- [LLM-QAT: Data-Free Quantization Aware Training for Large Language Models](https://mlsanity.com/paper/2305.17888/) (2023)
- [Deep multi-scale video prediction beyond mean square error](https://mlsanity.com/paper/1511.05440/) (2015)
- [A Survey on Knowledge Distillation of Large Language Models](https://mlsanity.com/paper/2402.13116/) (2024)
- [A Survey on Model Compression for Large Language Models](https://mlsanity.com/paper/2308.07633/) (2023)
- [COIN: COmpression with Implicit Neural representations](https://mlsanity.com/paper/2103.03123/) (2021)
- [WIRE: Wavelet Implicit Neural Representations](https://mlsanity.com/paper/2301.05187/) (2023)
- [Personalized Federated Learning using Hypernetworks](https://mlsanity.com/paper/2103.04628/) (2021)
- [HyperStyle: StyleGAN Inversion with HyperNetworks for Real Image Editing](https://mlsanity.com/paper/2111.15666/) (2021)
- [Focal Frequency Loss for Image Reconstruction and Synthesis](https://mlsanity.com/paper/2012.12821/) (2020)
- [Language model compression with weighted low-rank factorization](https://mlsanity.com/paper/2207.00112/) (2022)
- [Recurrent Residual Convolutional Neural Network based on U-Net (R2U-Net) for Medical Image Segmentation](https://mlsanity.com/paper/1802.06955/) (2018)
- [A Fast Post-Training Pruning Framework for Transformers](https://mlsanity.com/paper/2204.09656/) (2022)
- [Improving Neural Network Quantization without Retraining using Outlier Channel Splitting](https://mlsanity.com/paper/1901.09504/) (2019)
- [Modulated Periodic Activations for Generalizable Local Functional Representations](https://mlsanity.com/paper/2104.03960/) (2021)
- [Graph HyperNetworks for Neural Architecture Search](https://mlsanity.com/paper/1810.05749/) (2018)
- [Structured Pruning for Deep Convolutional Neural Networks: A survey](https://mlsanity.com/paper/2303.00566/) (2023)
- [A Brief Review of Hypernetworks in Deep Learning](https://mlsanity.com/paper/2306.06955/) (2023)
- [Generative Low-bitwidth Data Free Quantization](https://mlsanity.com/paper/2003.03603/) (2020)

## Abstract (arXiv)

With the development of foundational models, model compression has become a critical requirement. Various model compression approaches have been proposed such as low-rank decomposition, pruning, quantization, ergodic dynamic systems, and knowledge distillation, which are based on different heuristics. To elevate the field from fragmentation to a principled discipline, we construct a unifying mathematical framework for model compression grounded in measure theory. We further demonstrate that each model compression technique is mathematically equivalent to a neural network subject to a regularization. Building upon this mathematical and structural equivalence, we propose an experimentally-verified data-free model compression framework, termed textitBig2Small, which translates Implicit Neural Representations (INRs) from data domain to the domain of network parameters. textitBig2Small trains compact INRs to encode the weights of larger models and reconstruct the weights during inference. To enhance reconstruction fidelity, we introduce Outlier-Aware Preprocessing to handle extreme weight values and a Frequency-Aware Loss function to preserve high-frequency details. Experiments on image classification and segmentation demonstrate that textitBig2Small achieves competitive accuracy and compression ratios compared to state-of-the-art baselines.