# The Surprising Effectiveness of Noise Pretraining for Implicit Neural Representations

arXiv: https://arxiv.org/abs/2603.29034 · HTML: https://mlsanity.com/paper/2603.29034/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 51 references that are in the map.
- First posted on arXiv 30 Mar 2026.

## Builds on (most cited first, 9 of 9)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [From data to functa: Your data point is a function and you can treat it like one](https://mlsanity.com/paper/2201.12204/) (2022)
- [Spatial Functa: Scaling Functa to ImageNet Classification and Generation](https://mlsanity.com/paper/2302.03130/) (2023)
- [Generalizable Implicit Neural Representations via Instance Pattern Composers](https://mlsanity.com/paper/2211.13223/) (2022)
- [ResFields: Residual Neural Fields for Spatiotemporal Signals](https://mlsanity.com/paper/2309.03160/) (2023)
- [Siamese SIREN: Audio Compression with Implicit Neural Representations](https://mlsanity.com/paper/2306.12957/) (2023)
- [Learning Video Representations without Natural Videos](https://mlsanity.com/paper/2410.24213/) (2024)
- [SIEDD: Shared-Implicit Encoder with Discrete Decoders](https://mlsanity.com/paper/2506.23382/) (2025)

## Built on by (most cited first, 1 of 1)

- [What Cohort INRs Encode and Where to Freeze Them](https://mlsanity.com/paper/2605.08298/) (2026)

## Abstract (arXiv)

The approximation and convergence properties of implicit neural representations (INRs) are known to be highly sensitive to parameter initialization strategies. While several data-driven initialization methods demonstrate significant improvements over standard random sampling, the reasons for their success -- specifically, whether they encode classical statistical signal priors or more complex features -- remain poorly understood. In this study, we explore this phenomenon through a series of experimental analyses leveraging noise pretraining. We pretrain INRs on diverse noise classes (e.g., Gaussian, Dead Leaves, Spectral) and measure their ability to both fit unseen signals and encode priors for an inverse imaging task (denoising). Our analyses on image and video data reveal a surprising finding: simply pretraining on unstructured noise (Uniform, Gaussian) dramatically improves signal fitting capacity compared to all other baselines. However, unstructured noise also yields poor deep image priors for denoising. In contrast, we also find that noise with the classic 1/|fα| spectral structure of natural images achieves an excellent balance of signal fitting and inverse imaging capabilities, performing on par with the best data-driven initialization methods. This finding enables more efficient INR training in applications lacking sufficient prior domain-specific data. For more details, visit project page at https://kushalvyas.github.io/noisepretraining.html