# Weight-Space Mixture-of-Experts for Implicit Neural Representation Classification

arXiv: https://arxiv.org/abs/2607.29463 · HTML: https://mlsanity.com/paper/2607.29463/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Vision mixture-of-experts, one of 12 directions in Mixture-of-experts.
- Builds on 40 of its 46 references that are in the map.
- First posted on arXiv 31 Jul 2026.

## Placed in

- Mixture-of-experts / [Vision mixture-of-experts](https://mlsanity.com/moe/vision-mixture-of-experts/)

## Builds on (most cited first, 30 of 40)

- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Implicit Neural Representations with Periodic Activation Functions](https://mlsanity.com/paper/2006.09661/) (2020)
- [Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains](https://mlsanity.com/paper/2006.10739/) (2020)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [RISE: Randomized Input Sampling for Explanation of Black-box Models](https://mlsanity.com/paper/1806.07421/) (2018)
- [Scaling Vision with Sparse Mixture of Experts](https://mlsanity.com/paper/2106.05974/) (2021)
- [Linear Mode Connectivity and the Lottery Ticket Hypothesis](https://mlsanity.com/paper/1912.05671/) (2019)
- [Interpretable Explanations of Black Boxes by Meaningful Perturbation](https://mlsanity.com/paper/1704.03296/) (2017)
- [Git Re-Basin: Merging Models modulo Permutation Symmetries](https://mlsanity.com/paper/2209.04836/) (2022)
- [Understanding Deep Networks via Extremal Perturbations and Smooth Masks](https://mlsanity.com/paper/1910.08485/) (2019)
- [A Survey on Mixture of Experts in Large Language Models](https://mlsanity.com/paper/2407.06204/) (2024)
- [From data to functa: Your data point is a function and you can treat it like one](https://mlsanity.com/paper/2201.12204/) (2022)
- [fastai: A Layered API for Deep Learning](https://mlsanity.com/paper/2002.04688/) (2020)
- [Synthesizing the preferred inputs for neurons in neural networks via deep generator networks](https://mlsanity.com/paper/1605.09304/) (2016)
- [Equivariant Architectures for Learning in Deep Weight Spaces](https://mlsanity.com/paper/2301.12780/) (2023)
- [A Structured Dictionary Perspective on Implicit Neural Representations](https://mlsanity.com/paper/2112.01917/) (2021)
- [Scaling Vision-Language Models with Sparse Mixture of Experts](https://mlsanity.com/paper/2303.07226/) (2023)
- [Permutation Equivariant Neural Functionals](https://mlsanity.com/paper/2302.14040/) (2023)
- [On the Symmetries of Deep Learning Models and their Internal Representations](https://mlsanity.com/paper/2205.14258/) (2022)
- [Deep Learning on Implicit Neural Representations of Shapes](https://mlsanity.com/paper/2302.05438/) (2023)
- [Graph Neural Networks for Learning Equivariant Representations of Neural Networks](https://mlsanity.com/paper/2403.12143/) (2024)
- [Spatial Functa: Scaling Functa to ImageNet Classification and Generation](https://mlsanity.com/paper/2302.03130/) (2023)
- [Graph Metanetworks for Processing Diverse Neural Architectures](https://mlsanity.com/paper/2312.04501/) (2023)

## Abstract (arXiv)

Implicit Neural Representations (INRs) encode signals as the weights of a coordinate-based neural network and have recently been proposed as an alternative domain for downstream learning. While promising, classification directly in weight space remains challenging due to the high dimensionality and complex structure of INR parameters. Furthermore, the way discriminative information is distributed across INR weights remains poorly understood. We propose a hierarchical Mixture-of-Experts (HMoE) Transformer that processes INR weights using conditional computation aligned with the structure of the underlying implicit network. Coupled with a meta-learning framework that shapes INR parameters for downstream tasks, our model achieves state-of-the-art accuracy across standard benchmarks, ranging from low-resolution datasets to high-resolution ImageNet-1K. To gain insight into how INRs encode discriminative information, we develop weight-space attribution and pruning methods that identify parameters most relevant for classification. These analyses reveal how class-specific structure emerges within INR layers and support the suitability of MoE architectures for weight-space learning. Our approach advances both the performance and interpretability of weight-space classifiers.