# Enhancing Accuracy and Parameter-Efficiency of Neural Representations for Network Parameterization

arXiv: https://arxiv.org/abs/2407.00356 · HTML: https://mlsanity.com/paper/2407.00356/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 33 references that are in the map.
- First posted on arXiv 29 Jun 2024.

## Builds on (most cited first, 24 of 24)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016)
- [Averaging Weights Leads to Wider Optima and Better Generalization](https://mlsanity.com/paper/1803.05407/) (2018)
- [Editing Models with Task Arithmetic](https://mlsanity.com/paper/2212.04089/) (2022)
- [SNIP: Single-shot Network Pruning based on Connection Sensitivity](https://mlsanity.com/paper/1810.02340/) (2018)
- [Merging Models with Fisher-Weighted Averaging](https://mlsanity.com/paper/2111.09832/) (2021)
- [Rethinking the Value of Network Pruning](https://mlsanity.com/paper/1810.05270/) (2018)
- [INeRF: Inverting Neural Radiance Fields for Pose Estimation](https://mlsanity.com/paper/2012.05877/) (2020)
- [Knowledge distillation: A good teacher is patient and consistent](https://mlsanity.com/paper/2106.05237/) (2021)
- [Online Knowledge Distillation with Diverse Peers](https://mlsanity.com/paper/1912.00350/) (2019)
- [The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction](https://mlsanity.com/paper/2312.13558/) (2023)
- [Structured Pruning for Deep Convolutional Neural Networks: A survey](https://mlsanity.com/paper/2303.00566/) (2023)
- [Convolutional Neural Network Pruning with Structural Redundancy Reduction](https://mlsanity.com/paper/2104.03438/) (2021)
- [Diffusion-Based Neural Network Weights Generation](https://mlsanity.com/paper/2402.18153/) (2024)
- [Can We Scale Transformers to Predict Parameters of Diverse ImageNet Models?](https://mlsanity.com/paper/2303.04143/) (2023)
- [Stochastic Weight Averaging Revisited](https://mlsanity.com/paper/2201.00519/) (2022)
- [NeRN -- Learning Neural Representations for Neural Networks](https://mlsanity.com/paper/2212.13554/) (2022)
- [PELA: Learning Parameter-Efficient Models with Low-Rank Approximation](https://mlsanity.com/paper/2310.10700/) (2023)
- [Capturing dynamical correlations using implicit neural representations](https://mlsanity.com/paper/2304.03949/) (2023)

## Built on by (most cited first, 2 of 2)

- [A Middle Path for On-Premises LLM Deployment: Preserving Privacy Without Sacrificing Model Confidentiality](https://mlsanity.com/paper/2410.11182/) (2024)
- [SBS: Enhancing Parameter-Efficiency of Neural Representations for Neural Networks via Spectral Bias Suppression](https://mlsanity.com/paper/2509.07373/) (2025)

## Abstract (arXiv)

In this work, we investigate the fundamental trade-off regarding accuracy and parameter efficiency in the parameterization of neural network weights using predictor networks. We present a surprising finding that, when recovering the original model accuracy is the sole objective, it can be achieved effectively through the weight reconstruction objective alone. Additionally, we explore the underlying factors for improving weight reconstruction under parameter-efficiency constraints, and propose a novel training scheme that decouples the reconstruction objective from auxiliary objectives such as knowledge distillation that leads to significant improvements compared to state-of-the-art approaches. Finally, these results pave way for more practical scenarios, where one needs to achieve improvements on both model accuracy and predictor network parameter-efficiency simultaneously.