# Meta-Learning Sparse Compression Networks

arXiv: https://arxiv.org/abs/2205.08957 · HTML: https://mlsanity.com/paper/2205.08957/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 54 references that are in the map.
- First posted on arXiv 18 May 2022.

## Builds on (most cited first, 30 of 38)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [Efficient Geometry-aware 3D Generative Adversarial Networks](https://mlsanity.com/paper/2112.07945/) (2021)
- [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021)
- [The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables](https://mlsanity.com/paper/1611.00712/) (2016)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning](https://mlsanity.com/paper/1711.05769/) (2017)
- [Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations](https://mlsanity.com/paper/1906.01618/) (2019)
- [Learning to learn by gradient descent by gradient descent](https://mlsanity.com/paper/1606.04474/) (2016)
- [Learning Sparse Neural Networks through L0 Regularization](https://mlsanity.com/paper/1712.01312/) (2017)
- [Rigging the Lottery: Making All Tickets Winners](https://mlsanity.com/paper/1911.11134/) (2019)
- [The State of Sparsity in Deep Neural Networks](https://mlsanity.com/paper/1902.09574/) (2019)
- [Meta-SGD: Learning to Learn Quickly for Few-Shot Learning](https://mlsanity.com/paper/1707.09835/) (2017)
- [Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Science](https://mlsanity.com/paper/1707.04780/) (2017)
- [NeRV: Neural Representations for Videos](https://mlsanity.com/paper/2110.13903/) (2021)
- [Learned Initializations for Optimizing Coordinate-Based Neural Representations](https://mlsanity.com/paper/2012.02189/) (2020)
- [COIN: COmpression with Implicit Neural representations](https://mlsanity.com/paper/2103.03123/) (2021)
- [MetaSDF: Meta-learning Signed Distance Functions](https://mlsanity.com/paper/2006.09662/) (2020)
- [Implicit Neural Representations for Image Compression](https://mlsanity.com/paper/2112.04267/) (2021)
- [Modulated Periodic Activations for Generalizable Local Functional Representations](https://mlsanity.com/paper/2104.03960/) (2021)
- [Meta-Learning with Warped Gradient Descent](https://mlsanity.com/paper/1909.00025/) (2019)
- [One Model To Learn Them All](https://mlsanity.com/paper/1706.05137/) (2017)

## Built on by (most cited first, 22 of 22)

- [Spatial Functa: Scaling Functa to ImageNet Classification and Generation](https://mlsanity.com/paper/2302.03130/) (2023)
- [C3: High-performance and low-complexity neural compression from a single image or video](https://mlsanity.com/paper/2312.02753/) (2023)
- [Hypernetwork-based Meta-Learning for Low-Rank Physics-Informed Neural Networks](https://mlsanity.com/paper/2310.09528/) (2023)
- [Online Adaptation of Language Models with a Memory of Amortized Contexts](https://mlsanity.com/paper/2403.04317/) (2024)
- [Compressing multidimensional weather and climate data into neural networks](https://mlsanity.com/paper/2210.12538/) (2022)
- [Compression with Bayesian Implicit Neural Representations](https://mlsanity.com/paper/2305.19185/) (2023)
- [Modality-Agnostic Variational Compression of Implicit Neural Representations](https://mlsanity.com/paper/2301.09479/) (2023)
- [Learning Large-scale Neural Fields via Context Pruned Meta-Learning](https://mlsanity.com/paper/2302.00617/) (2023)
- [Good, Cheap, and Fast: Overfitted Image Compression with Wasserstein Distortion](https://mlsanity.com/paper/2412.00505/) (2024)
- [Versatile Neural Processes for Learning Implicit Neural Representations](https://mlsanity.com/paper/2301.08883/) (2023)
- [Meta-Learning with Self-Improving Momentum Target](https://mlsanity.com/paper/2210.05185/) (2022)
- [RECOMBINER: Robust and Enhanced Compression with Bayesian Implicit Neural Representations](https://mlsanity.com/paper/2309.17182/) (2023)
- [Latent-INR: A Flexible Framework for Implicit Representations of Videos with Discriminative Semantics](https://mlsanity.com/paper/2408.02672/) (2024)
- [L0onie: Compressing COINs with L0-constraints](https://mlsanity.com/paper/2207.04144/) (2022)
- [Improved Positional Encoding for Implicit Neural Representation based Compact Data Representation](https://mlsanity.com/paper/2311.06059/) (2023)
- [Unleashing the Power of Meta-tuning for Few-shot Generalization Through Sparse Interpolated Experts](https://mlsanity.com/paper/2403.08477/) (2024)
- [Predicting the Encoding Error of SIRENs](https://mlsanity.com/paper/2410.21645/) (2024)
- [Implicit Neural Compression of Point Clouds](https://mlsanity.com/paper/2412.10433/) (2024)
- [D'OH: Decoder-Only Random Hypernetworks for Implicit Neural Representations](https://mlsanity.com/paper/2403.19163/) (2024)
- [Neuromodulated Meta-Learning](https://mlsanity.com/paper/2411.06746/) (2024)
- [Meta-Sparsity: Learning Optimal Sparse Structures in Multi-task Networks through Meta-learning](https://mlsanity.com/paper/2501.12115/) (2025)
- [AIR: Amortized Image Reconstruction Framework for Self-Supervised Feed-Forward 2D Gaussian Splatting](https://mlsanity.com/paper/2605.20820/) (2026)

## Abstract (arXiv)

Recent work in Deep Learning has re-imagined the representation of data as functions mapping from a coordinate space to an underlying continuous signal. When such functions are approximated by neural networks this introduces a compelling alternative to the more common multi-dimensional array representation. Recent work on such Implicit Neural Representations (INRs) has shown that - following careful architecture search - INRs can outperform established compression methods such as JPEG (e.g. Dupont et al., 2021). In this paper, we propose crucial steps towards making such ideas scalable: Firstly, we employ state-of-the-art network sparsification techniques to drastically improve compression. Secondly, introduce the first method allowing for sparsification to be employed in the inner-loop of commonly used Meta-Learning algorithms, drastically improving both compression and the computational cost of learning INRs. The generality of this formalism allows us to present results on diverse data modalities such as images, manifolds, signed distance functions, 3D shapes and scenes, several of which establish new state-of-the-art results.