mlsanityData to 7 Oct 2026

Improved Generalization of Weight Space Networks via Augmentations

arXiv 2402.04081 ↗First posted 6 Feb 2024cs.LG cs.AIcited by 24

Not on the map. Why?

Abstract (arXiv)

Learning in deep weight spaces (DWS), where neural networks process the weights of other neural networks, is an emerging research direction, with applications to 2D and 3D neural fields (INRs, NeRFs), as well as making inferences about other types of neural networks. Unfortunately, weight space models tend to suffer from substantial overfitting. We empirically analyze the reasons for this overfitting and find that a key reason is the lack of diversity in DWS datasets. While a given object can be represented by many different weight configurations, typical INR training sets fail to capture variability across INRs that represent the same object. To address this, we explore strategies for data augmentation in weight spaces and propose a MixUp method adapted for weight spaces. We demonstrate the effectiveness of these methods in two setups. In classification, they improve performance similarly to having up to 10 times more data. In self-supervised contrastive learning, they yield substantial 5-10% gains in downstream classification.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 28 of its 36 references that are in the map. First posted on arXiv 6 Feb 2024.

Builds on

28 of 36 references.

References · 28

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  2. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  3. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  4. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  5. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
  6. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
  7. 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,220
  8. 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,214
  9. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
  10. 2017Decoupled Weight Decay Regularizationcited by 1,722
  11. 2020Principal Neighbourhood Aggregation for Graph Netscited by 477
  12. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
16 more
  1. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285
  2. 2018Point Convolutional Neural Networks by Extension Operatorscited by 154
  3. 2022G-Mixup: Graph Data Augmentation for Graph Classificationcited by 145
  4. 2020Predicting Neural Network Accuracy from Weightscited by 107
  5. 2023Equivariant Architectures for Learning in Deep Weight Spacescited by 106
  6. 2020PointMixup: Augmentation for Point Cloudscited by 104
  7. 2020Self-Supervised Learning for Domain Adaptation on Point-Cloudscited by 99
  8. 2023Permutation Equivariant Neural Functionalscited by 69
  9. 2020Classifying the classifier: dissecting the weight space of neural networkscited by 62
  10. 2022Re-basin via implicit Sinkhorn differentiationcited by 61
  11. 2023Graph Metanetworks for Processing Diverse Neural Architecturescited by 55
  12. 2023Neural Functional Transformerscited by 50
  13. 2022Relational Attention: Generalizing Transformers for Graph-Structured Taskscited by 41
  14. 2023Equivariant Deep Weight Space Alignmentcited by 34
  15. 2023Graph Mixup with Soft Alignmentscited by 15
  16. 2023Set-based Neural Network Encoding Without Weight Tyingcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 11; 2026: 8.

24 papers.

Sort:

By area:

  1. 2025Symmetry in Neural Network Parameter Spacescited by 31
  2. 2025We Should Chart an Atlas of All the World's Modelscited by 25
  3. 2024The Empirical Impact of Neural Parameter Symmetries, or Lack Thereofcited by 21
  4. 2024Learning on LoRAs: GL-Equivariant Processing of Low-Rank Weight Spaces for Large Finetuned Modelscited by 21
  5. 2024Deep Linear Probe Generators for Weight Space Learningcited by 17
  6. 2025End-to-End Implicit Neural Representations for Classificationcited by 7
  7. 2024Connecting NeRFs, Images, and Textcited by 6
  8. 2025Structure Is Not Enough: Leveraging Behavior for Neural Network Weight Reconstructioncited by 5
  9. 2024Foldable SuperNets: Scalable Merging of Transformers with Different Initializations and Taskscited by 4
  10. 2025Learning Model Representations Using Publicly Available Model Hubscited by 4
  11. 2025ARC: Anchored Representation Clouds for High-Resolution INR Classificationcited by 3
  12. 2025SINR: Sparsity Driven Compressed Implicit Neural Representationscited by 3
12 more
  1. 2025A Model Zoo on Phase Transitions in Neural Networkscited by 3
  2. 2026DeepWeightFlow: Re-Basined Flow Matching for Generating Neural Network Weightscited by 3
  3. 2026Position: Weight Space Should Be a First-Class Generative AI Modalitycited by 2
  4. 2025The Impact of Model Zoo Size and Composition on Weight Space Learningcited by 1
  5. 2026Dynamic Neural Graph Encoding of Inference Processes in Deep Weight Spacecited by 1
  6. 2025On the Internal Representations of Graph Metanetworkscited by 0
  7. 2025Metanetworks as Regulatory Operators: Learning to Edit for Requirement Compliancecited by 0
  8. 2026Discovering Hidden Gems in Model Repositoriescited by 0
  9. 2026Is Class Signal Clustered or Routed in Task-Induced Implicit Neural Representation Weight Spaces?cited by 0
  10. 2026Beyond Structural Symmetries: Linear Mode Connectivity via Neuron Identifiabilitycited by 0
  11. 2026Permutation-Equivariant Flow Matching for Alignment-Free Neural Weight Generationcited by 0
  12. 2026CrossGMN: Graph Metanetworks for Cross-Architecture Weight-Space Transformationscited by 0