mlsanityData to 7 Oct 2026

Equivariant Deep Weight Space Alignment

arXiv 2310.13397 ↗First posted 20 Oct 2023cs.LGcited by 34

Not on the map. Why?

Abstract (arXiv)

Permutation symmetries of deep networks make basic operations like model merging and similarity estimation challenging. In many cases, aligning the weights of the networks, i.e., finding optimal permutations between their weights, is necessary. Unfortunately, weight alignment is an NP-hard problem. Prior research has mainly focused on solving relaxed versions of the alignment problem, leading to either time-consuming methods or sub-optimal solutions. To accelerate the alignment process and improve its quality, we propose a novel framework aimed at learning to solve the weight alignment problem, which we name Deep-Align. To that end, we first prove that weight alignment adheres to two fundamental symmetries and then, propose a deep architecture that respects these symmetries. Notably, our framework does not require any labeled data. We provide a theoretical analysis of our approach and evaluate Deep-Align on several types of network architectures and learning setups. Our experimental results indicate that a feed-forward pass with Deep-Align produces better or equivalent alignments compared to those produced by current optimization algorithms. Additionally, our alignments can be used as an effective initialization for other methods, leading to improved solutions with a significant speedup in convergence.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Builds on 38 of its 52 references that are in the map. First posted on arXiv 20 Oct 2023.

Builds on

38 of 52 references.

References · 38

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  2. 2017Deep Setscited by 1,394
  3. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  4. 2022Editing Models with Task Arithmeticcited by 1,076
  5. 2018Machine Learning for Combinatorial Optimization: a Methodological Tour d'Horizoncited by 612
  6. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  7. 2020Federated Learning with Matched Averagingcited by 564
  8. 2017Learning Combinatorial Optimization Algorithms over Graphscited by 557
  9. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  10. 2018Invariant and Equivariant Graph Networkscited by 339
  11. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 328
  12. 2018On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groupscited by 296
26 more
  1. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285
  2. 2021Combinatorial optimization and reasoning with graph neural networkscited by 267
  3. 2018Learning a SAT Solver from Single-Bit Supervisioncited by 198
  4. 2023ZipIt! Merging Models from Different Tasks without Trainingcited by 178
  5. 2022Sign and Basis Invariant Networks for Spectral Graph Representation Learningcited by 176
  6. 2018Learning Latent Permutations with Gumbel-Sinkhorn Networkscited by 172
  7. 2017Equivariance Through Parameter-Sharingcited by 139
  8. 2022REPAIR: REnormalizing Permuted Activations for Interpolation Repaircited by 119
  9. 2020Predicting Neural Network Accuracy from Weightscited by 107
  10. 2023Equivariant Architectures for Learning in Deep Weight Spacescited by 106
  11. 2016Topology and Geometry of Half-Rectified Network Optimizationcited by 102
  12. 2020On Learning Sets of Symmetric Elementscited by 97
  13. 2020Deep Graph Matching Consensuscited by 85
  14. 2020Optimizing Mode Connectivity via Neuron Alignmentcited by 85
  15. 2020Learning Combinatorial Optimization on Graphs: A Survey with Applications to Networkingcited by 70
  16. 2023Permutation Equivariant Neural Functionalscited by 69
  17. 2018Deep Models of Interactions Across Setscited by 68
  18. 2020Classifying the classifier: dissecting the weight space of neural networkscited by 62
  19. 2022Re-basin via implicit Sinkhorn differentiationcited by 61
  20. 2023Neural Functional Transformerscited by 50
  21. 2017Revised Note on Learning Algorithms for Quadratic Assignment with Graph Neural Networkscited by 28
  22. 2022Wasserstein Barycenter-based Model Fusion and Linear Mode Connectivity of Neural Networkscited by 14
  23. 2020Equivariant Maps for Hierarchical Structurescited by 11
  24. 2016Exact Recovery with Symmetries for Procrustes Matchingcited by 10
  25. 2023Set-based Neural Network Encoding Without Weight Tyingcited by 9
  26. 2017Exact Recovery with Symmetries for the Doubly-Stochastic Relaxationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 16; 2025: 13; 2026: 5.

34 papers.

Sort:

By area:

  1. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  2. 2025Symmetry in Neural Network Parameter Spacescited by 31
  3. 2025We Should Chart an Atlas of All the World's Modelscited by 25
  4. 2024Improved Generalization of Weight Space Networks via Augmentationscited by 24
  5. 2024Scale Equivariant Graph Metanetworkscited by 23
  6. 2024The Empirical Impact of Neural Parameter Symmetries, or Lack Thereofcited by 21
  7. 2024Learning on LoRAs: GL-Equivariant Processing of Low-Rank Weight Spaces for Large Finetuned Modelscited by 21
  8. 2024Deep Linear Probe Generators for Weight Space Learningcited by 17
  9. 2024PLeaS -- Merging Models with Permutations and Least Squarescited by 16
  10. 2025Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusioncited by 16
  11. 2024Unsupervised Model Tree Heritage Recoverycited by 11
  12. 2025Scalable Model Merging with Progressive Layer-wise Distillationcited by 9
22 more
  1. 2025The Butterfly Effect: Neural Network Training Trajectories Are Highly Sensitive to Initial Conditionscited by 9
  2. 2024Rethinking Weight-Averaged Model-mergingcited by 8
  3. 2024Learning on Model Weights using Tree Expertscited by 7
  4. 2025End-to-End Implicit Neural Representations for Classificationcited by 7
  5. 2025GradMetaNet: An Equivariant Architecture for Learning on Gradientscited by 7
  6. 2024Variational Inference Failures Under Model Symmetries: Permutation Invariant Posteriors for Bayesian Neural Networkscited by 6
  7. 2025Structure Is Not Enough: Leveraging Behavior for Neural Network Weight Reconstructioncited by 5
  8. 2024Foldable SuperNets: Scalable Merging of Transformers with Different Initializations and Taskscited by 4
  9. 2024Vanishing Feature: Diagnosing Model Merging and Beyondcited by 3
  10. 2025A Model Zoo on Phase Transitions in Neural Networkscited by 3
  11. 2024Revisiting Multi-Permutation Equivariance through the Lens of Irreducible Representationscited by 2
  12. 2024Exploring Model Kinship for Merging Large Language Modelscited by 2
  13. 2025Accelerating Vehicle Routing via AI-Initialized Genetic Algorithmscited by 2
  14. 2025Cross-Model Semantics in Representation Learningcited by 2
  15. 2025The Impact of Model Zoo Size and Composition on Weight Space Learningcited by 1
  16. 2026Ensuring Semantics in Weights of Implicit Neural Representations through the Implicit Function Theoremcited by 1
  17. 2024Wolf2Pack: The AutoFusion Framework for Dynamic Parameter Fusioncited by 0
  18. 2025Symmetry-Aware Graph Metanetwork Autoencoders: Model Merging through Parameter Canonicalizationcited by 0
  19. 2026Is Class Signal Clustered or Routed in Task-Induced Implicit Neural Representation Weight Spaces?cited by 0
  20. 2026Beyond Structural Symmetries: Linear Mode Connectivity via Neuron Identifiabilitycited by 0
  21. 2026Task-Restricted Symmetries in Recurrent Weight Spacecited by 0
  22. 2026Permutation-Equivariant Flow Matching for Alignment-Free Neural Weight Generationcited by 0