mlsanityData to 7 Oct 2026

Finite Probes Suffice: Identifiability and Universality for Weight-Space Learning

arXiv 2609.33901 ↗First posted 27 Sep 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Learning properties of neural networks has recently attracted growing interest, with existing approaches operating either directly on network parameters or through probe-based representations of network behavior. While probing methods have shown strong empirical performance, their theoretical foundations remain limited. In this work, we study when finite probe-based representations are sufficient for learning neural functionals. We establish general identification and universality results for probing, and show that using intermediate hidden representations can provide significantly more informative representations than relying only on final outputs. Motivated by these results, we introduce HIDDENPROBE, a simple architecture for learning from hidden probe responses. Across a range of neural functional benchmarks, including both MLPs and Transformers, HIDDENPROBE consistently improves over existing probing methods and achieves state-of-the-art performance. Our code is publicly available on GitHub.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 31 of its 42 references that are in the map. First posted on arXiv 27 Sep 2026.

Builds on

31 of 42 references.

References · 31

  1. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,120
  2. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  3. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  4. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  5. 2017Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networkscited by 304
  6. 2019On the equivalence between graph isomorphism testing and function approximation with GNNscited by 172
  7. 2019Complexity of Linear Regions in Deep Networkscited by 166
  8. 2020Predicting Neural Network Accuracy from Weightscited by 107
  9. 2023Equivariant Architectures for Learning in Deep Weight Spacescited by 106
  10. 2023Permutation Equivariant Neural Functionalscited by 69
  11. 2024Graph Neural Networks for Learning Equivariant Representations of Neural Networkscited by 59
  12. 2023Graph Metanetworks for Processing Diverse Neural Architecturescited by 55
19 more
  1. 2023Neural Functional Transformerscited by 50
  2. 2021On the Optimal Memorization Power of ReLU Neural Networkscited by 40
  3. 2022Low Dimensional Invariant Embeddings for Universal Geometric Learningcited by 38
  4. 2023Neural Injective Functions for Multisets, Measures and Graphs via a Finite Witness Theoremcited by 33
  5. 2024Scale Equivariant Graph Metanetworkscited by 23
  6. 2024Equivariant Neural Functional Networks for Transformerscited by 20
  7. 2024Deep Linear Probe Generators for Weight Space Learningcited by 17
  8. 2024Monomial Matrix Group Equivariant Neural Functional Networkscited by 15
  9. 2024Learning Useful Representations of Recurrent Neural Network Weight Matricescited by 13
  10. 2024Equivariant Polynomial Functional Networkscited by 10
  11. 2023Set-based Neural Network Encoding Without Weight Tyingcited by 9
  12. 2023Analyzing Populations of Neural Networks via Dynamical Model Embeddingcited by 7
  13. 2024Learning on Model Weights using Tree Expertscited by 7
  14. 2026On the Expressive Power of Permutation-Equivariant Weight-Space Networkscited by 3
  15. 2024Revisiting Multi-Permutation Equivariance through the Lens of Irreducible Representationscited by 2
  16. 2026Provably Learning Attention with Queriescited by 1
  17. 2026Quasi-Equivariant Metanetworkscited by 1
  18. 2026What Linear Probes Miss: Multi-View Probing for Weight-Space Learningcited by 1
  19. 2026Dynamic Neural Graph Encoding of Inference Processes in Deep Weight Spacecited by 1

Built on by

Not cited yet.