mlsanityData to 13 Oct 2026

Optimizing Rank for High-Fidelity Implicit Neural Representations

arXiv 2512.14366 ↗First posted 16 Dec 2025cs.CVcited by 7

Not on the map. Why?

Abstract (arXiv)

Implicit Neural Representations (INRs) based on vanilla Multi-Layer Perceptrons (MLPs) are widely believed to be incapable of representing high-frequency content. This has directed research efforts towards architectural interventions, such as coordinate embeddings or specialized activation functions, to represent high-frequency signals. In this paper, we challenge the notion that the low-frequency bias of vanilla MLPs is an intrinsic, architectural limitation to learn high-frequency content, but instead a symptom of stable rank degradation during training. We empirically demonstrate that regulating the network's rank during training substantially improves the fidelity of the learned signal, rendering even simple MLP architectures expressive. Extensive experiments show that using optimizers like Muon, with high-rank, near-orthogonal updates, consistently enhances INR architectures even beyond simple ReLU MLPs. These substantial improvements hold across a diverse range of domains, including natural and medical images and novel view synthesis, with up to +9 dB PSNR over the same architecture. Code is available at (https://rank-inrs.github.io).

Key facts. Cited by 7 papers in the map (data through 13 Oct 2026). Builds on 69 of its 90 references that are in the map. First posted on arXiv 16 Dec 2025.

Builds on

69 of 90 references.

References · 69

  1. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,082
  2. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,712
  3. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,686
  4. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,610
  5. 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,225
  6. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,183
  7. 2021Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fieldscited by 1,958
  8. 2018On the Spectral Bias of Neural Networkscited by 1,329
  9. 2020Implicit Geometric Regularization for Learning Shapescited by 708
  10. 2021RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputscited by 614
  11. 2021Neural Fields in Visual Computing and Beyondcited by 539
  12. 2023Symbolic Discovery of Optimization Algorithmscited by 425
38 more
  1. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 413
  2. 2025Muon is Scalable for LLM Trainingcited by 406
  3. 2021NeRV: Neural Representations for Videoscited by 256
  4. 2020Learned Initializations for Optimizing Coordinate-Based Neural Representationscited by 241
  5. 2021COIN: COmpression with Implicit Neural representationscited by 233
  6. 2023WIRE: Wavelet Implicit Neural Representationscited by 220
  7. 2020MetaSDF: Meta-learning Signed Distance Functionscited by 214
  8. 2018Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learningcited by 213
  9. 2020ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learningcited by 201
  10. 2022From data to functa: Your data point is a function and you can treat it like onecited by 186
  11. 2024Old Optimizer, New Norm: An Anthologycited by 183
  12. 2021Implicit Neural Representations for Image Compressioncited by 162
  13. 2023A mathematical perspective on Transformerscited by 161
  14. 2021The Low-Rank Simplicity Bias in Deep Networkscited by 152
  15. 2021BACON: Band-limited Coordinate Networks for Multiscale Scene Representationcited by 142
  16. 2020Frequency Bias in Neural Networks for Input of Non-Uniform Densitycited by 140
  17. 2021Modulated Periodic Activations for Generalizable Local Functional Representationscited by 140
  18. 2022Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapsecited by 138
  19. 2021Beyond Periodicity: Towards a Unifying Framework for Activations in Coordinate-MLPscited by 135
  20. 2024Challenges in Training PINNs: A Loss Landscape Perspectivecited by 111
  21. 2023Equivariant Architectures for Learning in Deep Weight Spacescited by 106
  22. 2025The Polar Express: Optimal Matrix Sign Methods and Their Application to the Muon Algorithmcited by 106
  23. 2022VideoINR: Learning Video Implicit Neural Representation for Continuous Space-Time Super-Resolutioncited by 101
  24. 2021A Structured Dictionary Perspective on Implicit Neural Representationscited by 100
  25. 2022Compressible-composable NeRF via Rank-residual Decompositioncited by 92
  26. 2022Learning Smooth Neural Functions via Lipschitz Regularizationcited by 86
  27. 2022COIN++: Neural Compression Across Modalitiescited by 82
  28. 2023NeuRBF: A Neural Fields Representation with Adaptive Radial Basis Functionscited by 65
  29. 2023Factor Fields: A Unified Framework for Neural Fields and Beyondcited by 58
  30. 2023Transformers learn through gradual rank increasecited by 52
  31. 2022Rank Diminishing in Deep Neural Networkscited by 47
  32. 2023Deep Transformers without Shortcuts: Modifying Self-attention for Faithful Signal Propagationcited by 45
  33. 2024On the Role of Attention Masks and LayerNorm in Transformerscited by 45
  34. 2015Equilibrated adaptive learning rates for non-convex optimizationcited by 42
  35. 2019Stable Rank Normalization for Improved Generalization in Neural Networks and GANscited by 41
  36. 2024Where Do We Stand with Implicit Neural Representations? A Technical and Performance Surveycited by 41
  37. 2023Single-subject Multi-contrast MRI Super-resolution via Implicit Neural Representationscited by 32
  38. 2023INCODE: Implicit Neural Conditioning with Prior Knowledge Embeddingscited by 31

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2026: 7.

7 papers.

  1. 2026Benchmarking Optimizers for MLPs in Tabular Deep Learningcited by 3
  2. 2026Low-Rank-Modulated Functa: Exploring the Latent Space of Implicit Neural Representations for Interpretable Ultrasound Video Analysiscited by 1
  3. 2026Frequency-Decomposed INR for NIR-Assisted Low-Light RGB Image Denoisingcited by 0
  4. 2026What Cohort INRs Encode and Where to Freeze Themcited by 0
  5. 2026Recurrent Sinusoidal INRs for Efficient High-Fidelity Representationcited by 0
  6. 2026Fast Implicit Neural Light Field Representation via Geometric Decomposition and Multi-Resolution Low-Rank Featurescited by 0
  7. 2026SA-Profile: Automated Sulcus Angle Profiling from Super-Resolution MRIcited by 0